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The  Legislative  Audit  Committee 
of  the  Montana  State  Legislature: 

This  is  our  performance  audit  of  Automated  System  Development  and 
Maintenance  at  the  Department  of  Administration,  the  Department  of  Social  and 
Rehabilitation  Services,  and  the  Department  of  Labor  and  Industry.  This  report 
contains  recommendations  concerning  management  of  system  development  and 
maintenance  activities  within  the  three  agencies.   Department  responses  are 
contained  at  the  end  of  the  report. 

We  wish  to  express  our  appreciation  to  the  staff  of  each  of  the  departments 
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Introduction 


The  Legislative  Audit  Committee  requested  a  performance  audit 
of  automated  system  development  and  maintenance  activities  of 
the  Department  of  Administration's  Application  Services  Bureau 
(ASB).  Our  preliminary  review  indicated  the  need  to  extend  our 
performance  audit  to  other  state  agencies  due  to  the  amount  of 
system  development  and  maintenance  activity  outside  ASB.  This 
report  addresses  our  audit  of  automated  system  development  and 
maintenance  activities  within  three  state  agencies:  the 
Department  of  Administration,  the  Department  of  Social  and 
Rehabilitation  Services,  and  the  Department  of  Labor  and 
Industry. 


The  objectives  of  the  audit  included  identifying  and  evaluating 
controls  which  ensure  automated  systems  are  developed  and 
maintained  in  an  efficient  and  effective  manner.   We  compared 
agency  system  development  and  maintenance  methodologies  to 
industry  standards.   In  addition,  we  determined  whether  systems 
developed  and  maintained  within  the  agencies  are  operating  as 
intended. 


Automated  System 
Development  and 
Maintenance 


The  life  of  an  automated  system  can  be  divided  into  three  basic 
areas:  development,  operations,  and  maintenance.   Development 
pertains  to  the  design,  formation,  testing,  documentation,  and 
implementation  of  a  new  system.  Operations  are  the  day-to-day 
procedures  and  processes  that  keep  a  system  functioning. 
Maintenance  provides  upkeep  and  sustains  the  operation  of  a 
previously  developed  system. 


A  widely  used  and  proven  approach  in  system  development  is  to 
divide  work  into  phases  which  are  logical  and  manageable.   This 
approach  is  used  by  private  industry,  governments,  and  the  three 
agencies  in  our  review.  This  formalized  method  of  system 
development  is  known  as  the  system  development  life  cycle. 

Once  a  system  has  been  implemented,  development  is  complete 
and  operation  and  maintenance  begin.   Most  data  processing 
organizations  have  an  operations  section  to  control  day-to-day 
processing.  The  operations  section  is  responsible  for  running 
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jobs  submitted  by  system  users,  printing  reports,  and  monitoring 
and  reporting  system  problems. 

System  maintenance  can  be  divided  into  phases  similar  to  those 
for  system  development.   System  maintenance  can  include 
modifications,  enhancements,  error  correction,  and  general 
maintenance.   Each  of  the  three  agencies  we  reviewed  uses  some 
form  of  a  system  maintenance  life  cycle.   The  main  difference 
between  system  development  and  system  maintenance  is  the  size 
of  projects. 


Agency  Operations 


ASB  is  the  only  state  entity  which  provides  system  development 
and  maintenance  services  to  other  state  agencies.   Personnel  in 
state  agencies  other  than  those  within  ASB  also  conduct  system 
development  and  maintenance.   However,  staff  responsibilities 
for  system  development  and  maintenance  are  limited  to  the 
department  where  they  are  employed. 


The  Department  of  Administration  has  statutory  authority  to 
establish  policies  and  a  statewide  plan  for  the  operation  and 
development  of  data  processing  for  state  government.   The 
department  also  has  authority  for  control  and  coordination  of  the 
implementation  of  information  systems  in  state  government.   The 
department  has  involved  agencies  in  state  data  processing  by 
creating  the  Information  Technology  Advisory  Council  and 
working  with  the  Information  Technology  Managers  Group. 


Department  of 
Administration 


At  the  Department  of  Administration,  system  development  and 
maintenance  is  conducted  by  staff  within  the  Application 
Services  Bureau  (ASB).   ASB  charges  fees  for  services  to  cover 
costs.  The  system  development  life  cycle  established  by  ASB 
consists  of  three  phases:  preliminary  analysis,  prototyping,  and 
system  test  and  installation.   Agencies  also  contract  with  ASB  for 
system  maintenance  services.   Maintenance  includes  any 
activities  required  to  keep  a  system  operational  and  responsive. 
Services  provided  include:  production  recovery,  minor 
maintenance  and  enhancement,  and  consultation. 
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ASB  Related  Issues  During  our  review  at  ASB,  we  noted  areas  where  system 

development  and  maintenance  activities  were  operating  as 
described.   The  system  development  and  maintenance 
methodologies  established  by  ASB  appear  logical  and  compare  to 
industry  standards.   ASB  management  has  a  process  in  place  to 
periodically  update  these  methodologies.   Users  of  the  systems 
we  reviewed  at  ASB  are  satisfied  with  the  development  and 
maintenance  processes,  and  believe  the  systems  are  beneficial. 

System  maintenance,  for  the  systems  we  reviewed,  appears  to  be 
completed  in  a  timely  manner.   ASB  staff  appear  to  have 
backgrounds,  experience,  and  skills  necessary  to  develop  and 
maintain  automated  systems.   In  addition,  staff  are  kept  up-to- 
date  on  appropriate  technologies. 

We  did  note  an  area  where  improvements  could  be  made  at  ASB. 
The  following  section  summarizes  our  findings. 

Post-Implementation  Review 

Development  standards  suggest  a  post-implementation  review  as 
the  final  phase  of  a  system  development  life  cycle  methodology. 
A  post-implementation  review  can  be  used  to  evaluate  the 
effectiveness  of  the  development  process.   ASB  management  has 
not  implemented  any  formal  post-implementation  review. 

ASB  is  not  getting  formal  feedback  on  its  system  development 
process.  If  problems  occur  as  a  result  of  deficiencies  in  ASB 
development  techniques,  future  development  projects  may  have 
similar  problems.   Also,  if  agencies  are  not  satisfied  with  ASB 
services,  future  contracts  may  not  be  considered. 

The  other  two  agencies  included  in  our  review  have  established 
post-implementation  review  as  part  of  the  system  development 
process.  To  increase  the  effectiveness  of  current  methodologies 
and  ensure  system  quality,  ASB  management  should  develop  and 
implement  policies  for  reviewing  systems  and  agency 
satisfaction. 
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Department  of  Social  and 
Rehabilitation  Services 


The  Operations  and  Technology  Division  (OTD)  is  responsible 
for  system  development  and  maintenance  for  the  department. 
Two  bureaus  within  the  OTD  provide  system  development  and 
maintenance  services.   The  Internal  Mainframe  Systems  Bureau 
is  responsible  for  mainframe  development  and  maintenance.   The 
Microcomputer  Applications  Bureau  handles  microcomputer 
based  development  and  maintenance. 


OTD's  new  system  development  life  cycle  consists  of  the 
following  five  phases:  conceptual,  definition,  design, 
implementation,  and  evaluation/acceptance.   System  maintenance 
services  provided  include:  production  recovery,  maintenance, 
and  enhancement. 


OTD  Related  Issues 


During  our  review  at  OTD,  we  identified  areas  where  system 
development  and  maintenance  activities  were  operating  as 
described.  System  development  and  maintenance  methodologies 
have  been  established  by  management  and  compare  to  industry 
standards.   Users  of  the  systems  we  reviewed  at  OTD  are 
satisfied  with  the  development  and  maintenance  processes,  and 
believe  the  systems  are  beneficial. 


System  maintenance,  for  the  systems  we  reviewed,  appears  to  be 
completed  in  a  timely  manner.  Staff  appear  to  have  experience 
and  skills  necessary  to  develop  and  maintain  automated  systems. 

We  did  note  an  area  where  improvements  could  be  made.   The 
following  section  discusses  training  at  OTD  and  includes  our 
recommendation  for  improvement. 

Training 

Overall,  we  found  the  department  provides  only  minimal 
training  for  system  development  and  maintenance  staff.   SRS 
does  not  have  a  formal  training  program,  nor  does  it  formally 
track  employee  training.  Staff  receive  some  training  which 
appears  appropriate.   However,  the  amount  of  training  received 
does  not  appear  to  keep  staff  up-to-date  on  relevant  data 
processing  technology. 
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Training  is  provided  to  improve  or  enhance  employees'  abilities 
to  perform  job  duties  and  can  improve  employee  morale. 
Without  a  formal  training  program,  data  processing  staff  have 
less  formal  access  to  changing  standards  and/or  up-to-date 
development  and  maintenance  technologies.   We  believe  OTD 
management  should  identify  training  needs  and  implement  a 
formal  employee  training  program  to  meet  these  needs. 


Department  of  Labor  and 
Industry 


Currently,  three  divisions  within  the  Department  of  Labor  and 
Industry  (DOLI)  are  responsible  for  system  development  and 
maintenance:  Job  Service,  Unemployment  Insurance,  and 
Centralized  Services.   Staff  from  various  sections  within  these 
three  divisions  complete  system  development  and  maintenance 
activities. 


The  department's  system  development  life  cycle  consists  of  six 
phases:  project  definition,  analysis,  design,  coding,  testing,  and 
implementation.   System  maintenance  services  provided  include: 
production  recovery,  maintenance,  and  enhancement. 


DOLI  Related  Issues 


During  our  review  at  DOLI,  we  noted  areas  where  system 
development  and  maintenance  activities  were  operating  as 
described.   DOLI's  system  development  and  maintenance 
methodologies  appear  logical  and  compare  to  industry  standards. 
Users  of  the  systems  we  reviewed  are  satisfied  with  the 
development  and  maintenance  processes,  and  believe  the  systems 
are  beneficial. 


System  maintenance,  for  the  systems  we  reviewed,  appears  to  be 
completed  in  a  timely  manner.  Staff  appear  to  have  necessary 
skills  and  experience  to  develop  and  maintain  automated  systems. 
Staff  are  also  kept  up-to-date  on  appropriate  technologies. 

We  did  note  an  area  where  improvement  is  needed.   No  one 
entity  is  responsible  for  data  processing  activity  within  the 
department.  The  following  section  discusses  this  issue  and 
includes  our  recommendation  for  improvement. 
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Separation  of  Development/Maintenance  Staff 
Staff  within  three  divisions  of  DOLI  conduct  system 
development  and  maintenance.   No  single  entity  has 
input/control  over  development  and  maintenance  work 
conducted  by  personnel  within  DOLI.   There  are  differences  in 
the  way  system  development  and  maintenance  is  conducted 
within  different  divisions  of  the  department.   Department 
management  has  established  general  guidelines  for  department 
personnel,  but  there  is  no  formal  process  to  ensure  all  personnel 
follow  similar  standards. 

Development  of  standards  and  coordination  of  system 
development  and  maintenance  activities  should  help  reduce 
possible  duplication  of  effort.   In  addition,  department  standards 
and  coordination  should  help  maximize  consistency  and  system 
compatibility,  which  should  help  increase  information  sharing 
capabilities.   We  believe  the  department  should  define  roles  and 
responsibilities  for  all  DOLI  data  processing  personnel.   One 
entity  should  be  given  responsibility  for  direction  and  control  of 
data  processing  in  the  department,  including  development  of 
department-wide  standards. 


Common  Concerns 


All  three  agencies  included  in  our  review  have  established 
standards  for  developing  and  maintaining  automated  systems. 
We  found  personnel  have  necessary  skills  to  develop  and 
maintain  automated  systems.   We  did  note  several  system 
development  and  maintenance  concerns  common  to  all  three 
agencies  included  in  our  audit.   The  following  sections 
summarize  our  findings,  including  recommendations  for 
improvement.  Our  recommendations  relate  to  increasing  the 
effectiveness  of  current  operations. 


Coordination  in  System 
Development 


We  asked  data  processing  personnel  in  the  agencies  we  reviewed 
what  communication  and  coordination  they  have  with  other  state 
agencies.   We  found  very  little  communication  and  coordination 
between  state  agencies  regarding  system  development.   Potential 
benefits  may  never  be  realized  without  some  form  of 
communication  or  coordination.   As  a  result,  potential  cost 
savings,  increases  in  efficiency,  and  other  benefits  to  agencies 
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and  the  state  may  be  lost.   In  addition,  there  may  be  some 
duplication  of  effort  by  agencies  developing  similar  systems. 

The  Department  of  Administration  has  established  Montana 
Operations  Manual  (MOM)  policies  for  state  agencies  to  follow 
for  planning  and  managing  system  development  projects.   It  is 
the  responsibility  of  each  state  agency  to  develop  and  implement 
system  development  methodologies.   We  recommend  all  three 
agencies  incorporate  MOM  policy  regarding  coordination  and 
information  sharing  into  current  methodologies. 

System  Testing  and  Test  Testing  is  a  requirement  in  all  industry  standards  regarding 

Documentation  system  development  and  maintenance,  including  standards 

developed  by  the  three  agencies  in  our  review.   Standards  also 
exist  regarding  test  documentation.   Our  review  indicates 
noncompliance  with  current  testing  policies  in  all  three  agencies. 
Although  it  appears  staff  within  these  three  agencies  conduct 
some  testing  during  system  development  and  maintenance, 
appropriate  documentation  of  testing  is  not  formally  maintained. 
Due  to  the  lack  of  appropriate  test  documentation,  there  is  no 
assurance  required  testing  was  completed. 

All  areas  critical  to  a  system  may  not  be  tested  if  test  plans  are 
not  developed,  reviewed,  and  approved.   Failure  to  adequately 
test  significant  system  components  could  create  problems  during 
future  system  operations.   According  to  industry  guidelines, 
correcting  system  errors  and/or  deficiencies  after  a  system  is  in 
production  is  much  more  time  consuming  and  costly  than  if  done 
during  initial  development. 

If  test  documentation  is  not  maintained  and  reviewed, 
management  cannot  properly  evaluate  system  and  staff 
performance.   Without  documentation  of  test  results, 
management  also  cannot  adequately  determine  whether  a  system 
or  system  change  is  ready  for  implementation.   We  recommend 
all  three  agencies  develop  more  specific  guidelines  on  testing 
procedures  and  test  documentation  content  and  retention. 
Management  should  also  enforce  testing  and  test  documentation 
requirements. 
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System  Documentation 


We  compared  system  documentation  standards  of  the  three 
departments  to  industry  standards.   We  also  reviewed 
documentation  for  several  systems  developed  and/or  maintained 
by  each  of  the  agencies.  Current  documentation  standards  of 
each  department  are  comparable  to  industry  standards. 
However,  we  found  established  policies  and  procedures  are  not 
followed.   It  does  not  appear  all  required  system  documentation 
is  maintained  for  system  development  and  maintenance  projects. 
In  addition,  documentation  is  not  consistent  between  projects. 


Lack  of  documentation  is  a  common  problem  associated  with 
development  and  maintenance  of  automated  systems. 
Inconsistency  and/or  lack  of  documentation  can  create  problems 
and  inefficiencies  in  operations.   System  documentation  is 
critical  for  development  and  maintenance  of  automated  systems. 
Management  of  all  three  agencies  should  emphasize  the 
importance  of  system  documentation  to  the  overall  project.   In 
addition,  management  should  ensure  complete  documentation 
exists,  as  appropriate,  for  all  system  development  and 
maintenance  projects. 


Management  Information 


Accurate,  timely,  and  appropriate  management  information  is 
essential  for  providing  effective  control.   Industry  standards 
suggest  collection  and  analysis  of  numerous  types  of  management 
information  as  part  of  project  control.   ASB  management  uses  a 
project  control  software  package  to  monitor  system  development 
and  maintenance  projects.   Neither  OTD  nor  DOLI  have 
management  information  systems  to  monitor  system  development 
and  maintenance.   In  addition,  there  is  no  formal  process  in 
place  for  evaluating  performance  in  completing  work  requests  on 
time.   Management  does  not  analyze  time  or  costs  associated  with 
system  development  and  maintenance. 


If  management  information  is  inaccurate  or  appropriate 
information  is  not  collected,  program  operations  could  suffer. 
Lack  of  management  information  could  cause  management  to 
make  uninformed  decisions  which  negatively  affect  staff  and/or 
the  quality  of  systems.   We  believe  management  at  all  three 
agencies  should  review  current  project  management  software  for 
adequacy  of  monitoring  projects  and  generating  management 
information.  Once  reviewed,  necessary  action  should  be  taken  to 
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ensure  adequate  information  is  available  to  make  informed 
decisions.   Management  should  also  gather  information  for  use  in 
measuring  system  and  staff  productivity. 


Performance  Appraisals 


We  examined  employee  personnel  files  to  determine  whether 
performance  appraisals  are  completed  in  accordance  with  the 
Administrative  Rules  of  Montana.   Performance  appraisals  are 
completed  at  ASB.   We  found  only  one  of  eight  OTD  personnel 
files  reviewed  had  a  current  performance  appraisal.   In  addition, 
we  found  four  of  eight  DOLI  personnel  files  reviewed  did  not 
have  a  current  performance  appraisal. 


Section  2.21.6411,  ARM,  states  performance  appraisals  are  to  be 
completed  for  each  full-time  and  part-time  position  during 
established  appraisal  periods  of  not  more  than  one  year's 
duration.   Appraisals  provide  direction  and  motivation  for 
employees.   Failure  to  complete  periodic  performance  appraisals 
makes  it  difficult  for  management  to  monitor  and  document 
employee  productivity  and  program  effectiveness.   We 
recommend  OTD  and  DOLI  management  complete  timely 
appraisals  of  all  staff. 


System  Development  and 
Maintenance  Standards 


OTD  management  has  no  formal  process  for  reviewing  and 
updating  standards.   DOLI  also  has  no  formal  process  for 
reviewing  and  updating  system  development  and  maintenance 
standards.   Established  standards  provide  general  controls  over 
system  development  and  maintenance  activity.   Controls  direct 
data  processing  staff  on  procedures  which  must  be  followed 
when  developing  new  systems  and/or  changing  existing  systems. 
Standards  must  be  carefully  planned  and  continually  monitored 
to  ensure  effective  implementation. 


Lack  of  formal  and  up-to-date  standards  can  cause 
inconsistencies  in  operations  and  impede  staff  performance. 
Without  a  formal  standards  monitoring  process,  the  status  of 
system  development  and  maintenance  functions  may  not  be 
known.  Staff  also  may  not  be  developing  and  maintaining 
systems  in  the  most  efficient  and  effective  manner. 
Management  at  OTD  and  DOLI  need  to  formalize  a  process  to 
review  and  update  standards  on  a  regular  basis. 
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Introduction 


The  Legislative  Audit  Committee  requested  a  performance  audit 
of  automated  system  development  and  maintenance  activities  of 
the  Department  of  Administration's  Application  Services  Bureau 
(ASB).   We  conducted  a  preliminary  review  to  determine  the 
need  for  a  performance  audit  and  identify  potential  audit  issues. 
Our  preliminary  review  indicated  the  need  to  extend  our  perfor- 
mance audit  to  other  state  agencies  due  to  the  amount  of  system 
development  and  maintenance  activity  outside  ASB.   This  report 
addresses  our  audit  of  automated  system  development  and  main- 
tenance activities  within  three  state  agencies: 

1.  The  Department  of  Administration. 

2.  The  Department  of  Social  and  Rehabilitation  Services. 

3.  The  Department  of  Labor  and  Industry. 

These  agencies  were  selected  based  on  number  of  staff  involved 
with  system  development  and  maintenance. 


Audit  Objectives 


The  objectives  of  the  audit  were  to: 

1 .  Identify  and  evaluate  controls  which  ensure  automated 
systems  are  developed  and  maintained  in  an  efficient  and 
effective  manner. 

2.  Compare  agency  system  development  and  maintenance 
methodologies  to  industry  standards. 

3.  Determine  whether  systems  developed  and  maintained 
within  the  agencies  are  operating  as  intended. 

4.  Review  compliance  with  applicable  statutes,  rules,  and 
regulations  regarding  automated  system  development  and 
maintenance. 

5.  Determine  the  implementation  status  of  applicable  recom- 
mendations resulting  from  a  1984  audit  of  the  Systems 
Development  Bureau,  now  ASB,  Department  of  Administra- 
tion, conducted  by  the  Office  of  the  Legislative  Auditor. 
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Audit  Scope  and 
Methodology 


During  preliminary  audit  work,  we  interviewed  Application 
Services  Bureau  staff  and  reviewed  system  documentation  and 
management  information.   We  obtained  general  information 
regarding  system  development  and  maintenance  activities  con- 
ducted by  other  state  agencies.   We  interviewed  agency  personnel 
involved  directly  with  system  development  and  maintenance  to 
obtain  input  on  current  methodologies  and  procedures.   Based  on 
preliminary  work  we  selected  the  three  agencies  noted  previously 
for  our  performance  audit. 


The  audit  was  conducted  in  accordance  with  governmental 
auditing  standards  for  performance  audits.   Formal  audit  work 
was  directed  at  automated  system  development  and  maintenance 
methodologies  and  controls.   We  did  not  include  the  computer 
operations  function  of  each  agency  as  part  of  our  review.   We 
evaluated  current  agency  methodologies  to  determine  if  methods 
are  logical  and  comparable  to  industry  standards.   We  reviewed 
information  and  documentation  at  each  agency  and  compared 
our  observations  to  agency  methodologies.   In  addition,  we 
evaluated  current  agency  procedures  for  periodically  reviewing 
and  updating  system  development  and  maintenance  policies  and 
procedures. 

Staff  qualifications  were  reviewed  to  determine  if  personnel 
have  necessary  skills  to  develop  and  maintain  automated  systems. 
Staff  training  records  were  reviewed  to  determine  whether  staff 
are  kept  up-to-date  on  appropriate  technologies.   Procedures  for 
reviewing  work  and  supervising  staff  were  evaluated  to  deter- 
mine whether  system  development  and  maintenance  activities  are 
properly  monitored. 

We  selected  two  systems  developed  and  maintained  by  each 
agency  and  reviewed  documentation  for  each  system.   We 
compared  system  documentation  to  agency  requirements  to 
determine  if  staff  follow  established  policies  and  procedures.   We 
verified  documentation  of  approval  of  development  and  mainte- 
nance activities  by  agency  management  and  user  representatives. 
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We  compared  available  cost  and  time  estimations  to  actual  results 
as  part  of  our  evaluation  of  controls  over  the  system  develop- 
ment and  maintenance  processes.   We  reviewed  documentation  to 
determine  whether  development  and  maintenance  is  completed 
in  a  timely  manner.   If  possible,  we  reviewed  development  and 
maintenance  documentation  for  the  same  systems. 

We  obtained  input  from  users  of  the  systems.   We  asked  users  for 
input  on  their  involvement  with  system  development  and  main- 
tenance activities.   Users  were  asked  about  their  satisfaction  with 
department  procedures,  current  system  operations,  and  if  the 
systems  are  beneficial.   Our  audit  also  included  a  review  of 
training  and  documentation  provided  to  system  users. 

Audit  scope  included  a  review  of  management  controls  over 
contracted  system  development  and  maintenance  services  to 
ensure  agency  and  state  needs  were  met.   The  Department  of 
Social  and  Rehabilitation  Services  is  the  only  agency  of  the  three 
reviewed  currently  contracting  for  system  development  and 
maintenance  services. 

We  evaluated  management  information  to  determine  if  manage- 
ment had  sufficient  information  to  make  informed  decisions. 
We  reviewed  agency  procedures  for  monitoring  development, 
maintenance,  and  costs.   We  identified  formal  reports  generated 
by  department  management  and  determined  how  management 
used  these  reports  to  evaluate  project  success. 

Our  review  included  determining  whether  a  process  is  in  place  at 
each  agency  to  ensure  system  development  and/or  maintenance 
will  be  beneficial  to  the  state.   We  evaluated  staff  and  user  input 
and  information  obtained  during  our  review  to  aid  in  our 
determination. 

Finally,  we  reviewed  applicable  audit  recommendations  made  to 
the  Department  of  Administration's  Systems  Development 
Bureau  (now  called  ASB)  from  a  1984  audit  conducted  by  the 
Office  of  the  Legislative  Auditor.   Our  review  was  conducted  to 
determine  the  implementation  status  of  prior  recommendations. 


Page  3 


Chapter  I 
Introduction 


We  incorporated  applicable  recommendations  into  our  perfor- 
mance audit  work. 


Data  Limitations 


Government  auditing  standards  require  the  disclosure  of  any 
constraints  imposed  on  the  audit  approach  because  of  data 
limitations.   Documentation  kept  during  system  development  and 
maintenance  at  the  three  agencies  reviewed  is  not  complete  and 
is  inconsistent.  Types  of  limited  information  included  estimated 
and  actual  costs  and  time  frames,  development  and  maintenance 
project  success  factors,  and  measurement  of  system  efficiency 
and  effectiveness.   We  were  hindered  in  achieving  our  audit 
objectives  because  of  limited  documentation.   This  data  limita- 
tion also  adversely  affects  management  of  system  development 
and  maintenance  activities.   The  data  limitations  and  their 
effects  are  discussed  in  detail  in  Chapters  III  through  VI. 


Compliance 


We  reviewed  agency  compliance  with  statutes  and  administrative 
rules  relating  to  automated  system  development  and  maintenance 
operations.  Generally,  we  found  all  three  agencies  to  be  in 
compliance  with  state  laws  and  administrative  rules. 


We  identified  noncompliance  related  to  completion  of  perfor- 
mance appraisals  within  the  Department  of  Social  and  Rehabili- 
tation Services  and  the  Department  of  Labor  and  Industry.   This 
issue  is  discussed  further  in  Chapter  VI. 


Implementation  Status 
of  Prior  Audit 
Recommendations 


As  part  of  our  performance  audit,  we  reviewed  the  implementa- 
tion status  of  prior  audit  recommendations  made  in  the  1984 
Electronic  Data  Processing  Audit  (84DP-12)  of  the  Systems 
Development  Bureau  (SDB),  Department  of  Administration. 
SDB  has  since  been  renamed  the  Application  Services  Bureau 
(ASB).  The  following  table  summarizes  the  status  of  prior  audit 
recommendations. 
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Table   1 

Recommendation  Status 

840P-12 

(based  on  limited  follow-up) 


Implemented 

Not  Fully  Implemented 

Not  Reviewed  * 


9 

13 

4 


*   Recommendations  not  reviewed  due  to  being  outside 
the  scope  of  this  audit. 

Source:  Compiled  by  the  Office  of  the  Legislative  Auditor. 


No  action  was  taken  on  5  of  the  13  prior  audit  recommendations 
not  fully  implemented.   Four  of  these  five  prior  audit  recom- 
mendations are  addressed  under  recommendations  made  in 
Chapter  VI  of  this  performance  audit  report.  The  remaining 
audit  recommendation  is  no  longer  applicable. 


Issue  for  Further  Study 


During  our  performance  audit  we  identified  an  issue  which  we 
believe  warrants  further  study.   Our  audit  scope  did  not  include 
an  evaluation  of  the  efficiency  and  effectiveness  of  systems 
within  the  three  agencies  reviewed.   We  did  not  review  and 
evaluate  program  languages,  code  structures,  system  structures, 
system  responsiveness,  error  rates,  or  other  performance 
measures. 


Costs  for  system  development  and  maintenance  are  high, 
especially  if  services  are  contracted  out  to  the  private  sector. 
Controls  are  needed  to  ensure  systems  meet  requirements  and 
benefit  agency  operations  and  the  state.  Currently,  there  does 
not  appear  to  be  any  procedures  in  place  in  state  government  to 
conduct  cost  benefit  analyses  of  automated  systems.   We  believe 
this  type  of  review  is  necessary  to  ensure  systems  benefit  agency 
operations  and  the  state.  These  reviews  could  help  identify 
inefficiencies  in  automated  systems  and  prevent  unnecessary 
costs  for  systems  development  and  maintenance. 
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Introduction 


This  chapter  provides  an  overview  of  system  development  and 
maintenance.   It  describes  current  industry  techniques  and 
provides  general  background  information.   The  techniques 
described  in  this  chapter  are  generally  used  by  the  three  agencies 
in  our  review. 


Automated  System 
Development  and 
Maintenance 


The  life  of  an  automated  system  can  be  divided  into  three  basic 
areas:  1)  development,  2)  operations,  and  3)  maintenance  (some- 
times referred  to  as  support).   Development  pertains  to  the 
design,  formation,  testing,  documentation,  and  implementation 
of  a  new  system.   Operations  are  the  day-to-day  procedures  and 
processes  that  keep  a  system  functioning.   Maintenance  provides 
upkeep  and  sustains  the  operation  of  a  previously  developed 
system.  The  following  sections  discuss  these  three  areas  in 
further  detail. 


System  Development  Life 
Cycle  (SDLC) 


A  widely  used  and  proven  approach  in  system  development  is  to 
divide  work  into  phases  which  are  logical  and  manageable.   This 
approach  is  used  by  private  industry,  governments,  and  the  three 
agencies  in  our  review.   This  formalized  method  of  system 
development  is  known  as  the  system  development  life  cycle 
(SDLC).  The  SDLC  can  vary  from  agency  to  agency,  but  in 
general,  the  life  cycle  follows  six  phases. 
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Figure  1 


System  Development  Life  Cycle 
Traditional 


Initiation 


Definition 


Design 


Development 


Testing 


Operation  & 
Maintenance 


Source:  Compiled  by  the  Office  of  the  Legislative  Auditor  from  industry  standards. 


SDLC  Initiation 


The  life  cycle  starts  with  the  recognition  of  a  problem  or  identi- 
fication of  a  need  for  a  new  system  to  improve  current  opera- 
tions.  The  general  nature  and  scope  of  the  project  should  be 
clearly  defined  and  approved.   A  feasibility  study  is  completed 
and  includes  a  review  of  alternative  solutions  and  a  cost  benefit 
analysis.   If  the  feasibility  study  is  approved,  the  life  cycle 
continues. 


SDLC  Definition 


The  second  phase  involves  development  of  system  requirements 
to  describe  how  the  system  will  function.   This  phase  includes 
development  of  system  flowcharts,  program  narratives,  file 
layouts,  and  data  descriptions.   Requirements  for  how  the  system 
will  function  as  well  as  requirements  for  what  types  of  data  the 
system  will  accept  are  developed  during  this  phase. 


SDLC  Design 


In  the  third  phase,  the  design  specifications  for  implementing 
the  system  are  developed.   This  includes  the  design  of  the 
system,  subsystems,  and  individual  programs.   In  addition, 
complete  descriptions  of  all  data,  files,  and  how  the  data  will  be 
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stored  are  produced.   A  detailed  model  of  the  final  system  is 
produced  during  this  phase. 


SDLC  Development  and 
Testing 


The  fourth  phase  is  when  actual  programming  or  coding  is 
completed.   Design  and  program  specifications  are  converted  to 
computer  code.   Once  written,  program  code  is  tested.   When 
programs  are  functioning  properly,  subsystems  are  tested. 
Finally,  the  system  as  a  whole  is  tested.   The  system  is  further 
tested  by  the  user.   System  documentation  is  finalized  including 
test  results,  user  manuals,  and  operation  manuals.   If  necessary,  a 
conversion  plan  to  convert  from  the  old  system  to  the  new 
system  is  developed.   After  the  system  is  operational,  a  post- 
implementation  review  is  conducted  to  determine  if  user  needs 
and  system  requirements  were  met. 


SDLC  Operation  and 
Maintenance 


Once  the  system  is  developed  and  accepted  by  the  user,  the 
operation  and  maintenance  phase  begins.   Maintenance  is 
considered  the  final  and  ongoing  phase  of  an  SDLC. 


SDLC  Variations 


Various  modifications  of  the  traditional  SDLC  are  continually 
developed  and  used  by  system  development  organizations.   All 
three  agencies  within  our  review  use  a  variation  of  the 
traditional  SDLC.   In  the  traditional  SDLC,  one  phase  is 
completed  prior  to  starting  the  next  phase.   One  approach  to 
varying  the  SDLC  is  to  start  subsequent  phases  prior  to  comple- 
tion of  previous  phases.   The  following  figure  depicts  this  varia- 
tion of  the  SDLC. 
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Figure  2 
System  Development  Life  Cycle 


Variation 


Initiation  U  Definition 


Design 


Development 
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Operation  & 
Maintenance 


Source:   Compiled  by  the  Office  of  the  Legislative  Auditor  from  industry  standards. 


Prototyping 


Another  variation  of  the  SDLC  is  prototyping.   The  Application 
Services  Bureau  at  the  Department  of  Administration  uses  proto- 
typing. Staff  within  the  Office  of  Information  Services  at  the 
Department  of  Labor  and  Industry  can  use  prototyping  if  pre- 
ferred.  Prototyping  is  based  on  building  and  using  a  model  of  a 
system  during  the  SDLC.  The  initial  version  of  the  prototype  is 
an  outline  of  the  final  system.  This  "prototype"  is  modified 
during  each  phase  of  the  SDLC  until  a  fully  functional  system  is 
implemented.   Prototyping  increases  active  participation  by  the 
user  which  helps  assure  system  requirements  are  met. 


System  Operation 


Once  a  system  has  been  implemented  and  a  post-implementation 
review  conducted,  development  is  complete  and  operation  and 
maintenance  begin.   Most  data  processing  organizations  have  an 
operations  section  to  control  day-to-day  processing.   The  opera- 
tions section  is  responsible  for  running  jobs  submitted  by  system 
users,  printing  reports,  and  monitoring  and  reporting  system 
problems. 


Page  10 


Chapter  II 
Background 


System  Maintenance  Life 
Cycle 


System  maintenance  can  be  divided  into  phases  similar  to  those 
for  system  development.   System  maintenance  can  include  modi- 
fications, enhancements,  error  correction,  and  general  mainte- 
nance.  Each  of  the  three  agencies  we  reviewed  uses  some  form 
of  a  system  maintenance  life  cycle.   The  main  difference 
between  system  development  and  system  maintenance  is  the  size 
of  projects.   As  a  result,  system  maintenance  activities  are  often 
more  brief  and  direct.  The  following  figure  shows  a  typical 
system  maintenance  life  cycle. 


Figure  3 

System  Maintenance  Life  Cycle 
Typical 
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Source:    Compiled  by  the  Office  of  the  Legislative  Auditor  from  industry  standards. 


Record  and  Analyze 


In  the  first  phase,  user  requests  for  system  changes  are  recorded 
and  assigned  for  analysis.   The  change  is  then  analyzed  and 
approved  for  completion.  This  phase  is  similar  to  the  feasibility 
study  completed  during  system  development. 


Design/Code  and  Test 


The  third  phase  involves  designing  and  coding  the  change. 
Phase  three  is  similar  to  the  phases  of  the  SDLC  where  design 
and  coding  are  conducted.  Testing  and  training  take  place 
during  phase  four.  Procedures  for  testing  during  maintenance 
are  the  same  as  for  development. 
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Implement 


Finally,  the  completed  change  is  implemented  and  system  moni- 
toring continues.  The  entire  process  is  completed  for  additional 
modifications  and  enhancements. 


All  modifications  and  enhancements  to  a  system  should  be 
approved  prior  to  completion,  and  tested  and  documented  prior 
to  being  placed  into  production.   User  management  should 
periodically  evaluate  system  operations  to  identify  problems  and 
the  need  for  maintenance  or  development  of  a  new  system. 


Agency  Operations 


Many  state  agencies  have  staff  involved  with  system  develop- 
ment and  maintenance.   One  distinction  exists  within  state 
automated  system  development  and  maintenance  -  the  Applica- 
tion Services  Bureau  (ASB)  at  the  Department  of  Administration 
is  the  only  state  agency  which  provides  system  development  and 
maintenance  services  to  other  state  agencies.   In  addition,  the 
Department  of  Administration  is  statutorily  responsible  for 
establishing  policies  and  a  statewide  plan  for  the  operation  and 
development  of  data  processing  in  state  government.   The 
department  is  also  responsible  for  review  and  approval  of  all 
contracts  for  private  data  processing  services. 


Personnel  in  state  agencies  other  than  those  within  ASB  also 
conduct  system  development  and  maintenance.   However,  staff 
responsibilities  for  system  development  and  maintenance  are 
limited  to  the  department  where  they  are  employed.   For 
example,  staff  within  the  Data  Processing  Bureau  at  the  Depart- 
ment of  Social  and  Rehabilitation  Services  provide  system  devel- 
opment and  maintenance  services  for  the  department  only.  The 
following  table  lists  state  agencies  and  the  number  of  staff 
involved  with  system  development  and  maintenance.  These 
figures  are  based  on  preliminary  review  findings  from 
September  1992. 
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Table  2 

System  Development  and  Maintenance 

Agencies  and  FTE 

DEPARTMENT/AGENCY 

# 

OF  FTE 

Administration  (ASB) 

19 

Agriculture 

3 

Commerce 

3 

Corrections  &  Human  Services 

4 

Fami ly  Services 

6 

Fish,  Wildlife,  &  Parks 

8 

Governor's  Office 

1 

Health  &  Environmental  Sciences 

7 

Justice 

10 

Labor  &  Industry 

19 

Livestock 

2 

Montana  State  Library 

5 

Natural  Resources  &  Conservation 

4 

Office  of  Public  Instruction 

5 

Public  Service  Regulation 

2 

Revenue 

16 

Social  &  Rehabilitation  Services 

14 

State  Auditor's  Office 

1 

State  Lands 

1 

Transportation 

18 

TOTAL 

148 

Source:  Compiled  by  the  Office  of 

the 

Legislative 

Auditor. 

House  Joint  Resolu- 
tion 48 


House  Joint  Resolution  (HJR)  48  passed  during  the  1991  Legis- 
lative Session  directed  the  Legislative  Finance  Committee  to 
study  information  management  services  within  state  government. 
A  subcommittee  of  the  Legislative  Finance  Committee  and  the 
Legislative  Audit  Committee  was  formed  to  complete  the  study. 
As  part  of  its  mandate,  the  subcommittee  was  to  evaluate 
whether  the  Department  of  Administration  has  effectively 
undertaken  its  responsibilities  in  the  development,  application, 
and  management  of  data  processing  systems  for  state 
government. 


In  its  report  to  the  legislature  (November  6,  1992),  the  subcom- 
mittee made  several  recommendations  to  the  Department  of 
Administration  related  to  system  development  and  maintenance 
within  Montana.   Recommendations  included: 


Page  13 


Chapter  II 
Background 


--     develop  a  definition  for  data  processing  services  that 
ensures  the  department  will  review  major  or  costly  data 
processing  service  contracts. 

--     continue  with  and  refine  efforts  to  coordinate  with  other 
state  agencies  through  the  Data  Processing  Management 
Group  and  the  Data  Processing  Advisory  Council  (DPAC). 

--     explicitly  state  in  its  rate  development  process  and  budget 
request  anticipated  subsidies  among  computer-related 
services  and  clearly  articulate  under  what  conditions  an 
ongoing  subsidy  is  considered  appropriate. 

--     centrally  publish  computer  policies  by  compiling  a  summary 
of  policies  in  a  single  publication  and  incorporating  central 
policies  in  Montana  Operations  Manual. 

--     expand  its  role  in  agency  development  of  large  projects  by 
using  DPAC  as  a  focal  point  for  agency  discussions  and 
planning  for  large  systems  that  could  serve  several  agencies. 

The  department  concurred  with  all  recommendations  made  by 
the  subcommittee.  The  DPAC,  now  called  the  Information 
Technology  Advisory  Council,  also  supported  the  recommenda- 
tions made  by  the  subcommittee. 


Data  Processing 
Oversight 


Section  2-17-501(1),  MCA,  gives  the  Department  of  Adminis- 
tration authority  to  establish  policies  and  a  statewide  plan  for  the 
operation  and  development  of  data  processing  for  state  govern- 
ment.  Section  1-0210,  Montana  Operations  Manual  (MOM), 
further  defines  statutes  by  giving  the  department  authority  for 
control  and  coordination  of  the  implementation  of  information 
systems  in  state  government. 


Due  to  recent  changes  in  statute  and  recommendations  from 
various  committees,  including  the  HJR  48  subcommittee,  the 
Department  of  Administration  has  become  more  involved  in  data 
processing  in  Montana.   The  department  has  involved  agencies  in 
state  data  processing  by  creating  the  Information  Technology 
Advisory  Council  (ITAC)  and  working  with  the  Information 
Technology  Managers  Group  (ITMG).   Information  Services 
Division  (ISD),  Department  of  Administration,  encourages 
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interaction  and  information  sharing,  and  emphasizes 
communication  and  coordination  through  the  ITAC  and  the 
ITMG.   ISD  staff  encourage  agencies  to  discuss  system 
development  projects  and  plans  at  ITAC  and  ITMG  meetings.   In 
addition,  ISD  is  finalizing  an  updated  version  of  its  data 
processing  directions  and  guidelines  document  which  addresses 
system  development. 
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Introduction 


This  chapter  provides  background  information  on  system 
development  and  maintenance  activities  within  the  Application 
Services  Bureau  (ASB),  Information  Services  Division  (ISD),  at 
the  Department  of  Administration.   It  also  discusses  specific 
issues  related  only  to  ASB. 


Application  Services 
Bureau  (ASB) 


At  the  Department  of  Administration,  system  development  and 
maintenance  is  conducted  by  staff  within  the  Project  Develop- 
ment Section  (PDS)  of  the  Application  Services  Bureau.   PDS 
staff  also  provide  mainframe  system  development  and  mainte- 
nance services  to  other  state  agencies  as  requested.   The  follow- 
ing figure  shows  ISD's  current  organization. 


Figure  4 

Information  Services  Division  Organization 


Information  Services  Division 


Central  Computer 
Operations  Bureau 


Office  of  Policy,  Research, 
and  Development 


Application  Services 
Bureau 


Fnd  User  Computing 
Section 


Telecom/Network 
Services  Bureau 


Application  Sollware 
Support  Section 


Proicct  Development 
Section 


Source:    Compiled  by  the  Office  of  the  Legislative  Auditor  from  department  records. 


The  Application  Services  Bureau  is  authorized  34  full-time 
equivalent  (FTE)  with  21  FTE  allocated  to  the  Project  Develop- 
ment Section.  These  21  FTE  include  1  secretary,  3  supervisors, 
16  information  systems  specialists,  and  1  vacant  supervisor 
position. 

For  fiscal  year  1992-93,  ASB  had  nine  development  projects. 
Two  projects  were  completed  and  seven  remained  active.   In 
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addition,  ASB  had  12  support/maintenance  contracts.   During 
fiscal  years  1988-89  through  1992-93,  ASB  started  23  develop- 
ment/enhancement projects.   Of  these  23  projects,  20  were 
completed  and  3  were  stopped  prior  to  system  implementation. 
Income  and  expenses  for  ASB  for  the  past  four  fiscal  years  are  as 
follows. 


Table   3 


Application  Services  Bureau 
System  Development/Maintenance  Income  and  Expenses 


FY  1989-90   FY  1990-91    FY  1991-92    FY  1992-93 


Income 

Expenses 

Difference 


$649,485 

701,245 

$(51,760) 


$826,210 

811,866 

$  14,344 


$845,404 
892,587 


$798,044 
854,227 


$(47,183)   $(56,183) 


Source:  Application  Services  Bureau  records. 


PDS  charges  fees  for  services  to  cover  costs.   Service  charges  for 
fiscal  year  1992-93  were  $36  per  hour.   ISD's  cash  balance  is 
used  to  cover  differences  between  income  and  expenses.   This 
cash  balance  is  derived  from  mainframe  processing  charges. 
According  to  Statewide  Budgeting  and  Accounting  System 
information,  ISD's  unreserved  fund  balance  for  fiscal  year  1992- 
93  is  approximately  $3,902,985. 


ASB  System  Development 


The  system  development  life  cycle  (SDLC)  established  by  ASB 
consists  of  three  phases:  1)  preliminary  analysis,  2)  prototyping, 
and  3)  system  test  and  installation.   ASB's  development  process 
starts  with  a  request  from  a  customer  (an  agency). 


Preliminary  Analysis 


Since  agencies  contract  with  ASB  for  development  services,  ASB 
assumes  the  initial  phase  of  the  system  development  life  cycle, 
which  includes  a  feasibility  study  and  cost  benefit  analysis,  is 
completed  by  the  agency  prior  to  requesting  service.    ASB's 
involvement  in  the  process  starts  with  development  of  system 
requirements,  which  is  normally  the  second  phase  of  a  traditional 
system  development  life  cycle.   A  preliminary  analysis  of 
customer  needs  and  system  requirements  is  completed  and  a  cost 
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Prototyping  -  Three  Levels 


estimate  is  developed  by  ASB.   If  the  customer  approves  the 
preliminary  analysis,  the  project  proceeds  to  the  second  phase. 

During  the  second  phase,  system  requirements  are  refined  and  an 
operational  system  is  designed  and  developed.   The  project  is 
divided  into  three  levels,  and  each  level  or  prototype  adds  to  the 
previous  level.   After  completion  of  the  level  three  prototyping 
phase,  a  fully  functional  automated  system  exists. 

The  final  phase  involves  system  testing  and  installation  of  the 
system.   ASB  tests  the  system  internally  first,  then  turns  the 
system  over  to  the  customer  for  testing  and  approval.   Testing  is 
completed  to  verify  the  system  meets  customer  needs.   During 
testing,  the  majority  of  errors  and  discrepancies  should  be 
identified  and  addressed.   ASB's  final  step  is  implementation  of 
the  new  system.   The  system  is  placed  into  production  status 
when  approved  by  the  customer.   At  this  point  the  project  is 
complete. 


System  Test  and 
Installation 


ASB  System  Maintenance 


When  a  system  is  approved  by  the  customer,  ASB  is  no  longer 
responsible  for  the  system.   However,  agencies  can  contract  with 
ASB  for  system  maintenance  services.   A  service  agreement  is 
developed  and  approved  by  ASB  and  the  user  agency.   Service 
agreements  are  usually  one  year  in  length. 


Maintenance  includes  any  activities  required  to  keep  a  system 
operational  and  responsive.   Services  provided  include:  1) 
production  recovery,  2)  minor  maintenance  and  enhancement, 
and  3)  consultation.   Production  recovery  is  correction  of 
problems  to  maintain  proper  system  operation.   Minor  mainte- 
nance and  enhancements  include  system  changes  to  maintain 
consistent  operation  or  strengthen  the  system  in  some  fashion. 
Staff  correct  problems  and  complete  system  changes  as  necessary 
using  procedures  similar  to  those  used  for  development.   Consul- 
tation is  time  spent  with  agency  personnel  discussing  system 
related  subjects  for  a  system  under  a  service  agreement. 
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ASB  Contracted  Services 


We  did  not  review  current  procedures  for  contracted  services  at 
ASB.   The  Information  Services  Division  has  a  $200,000  biennial 
appropriation  for  consulting  services,  but  has  not  contracted  for 
system  development  or  maintenance  services  in  the  past  four 
fiscal  years,  1988-89  to  1991-92.   ISD  did  hire  consultants  to 
provide  other  types  of  software  support  during  this  same  time 
period  ($137,738  total). 


ASB  Related  Issues 


Introduction 


During  our  review  at  ASB,  we  noted  areas  where  system  devel- 
opment and  maintenance  activities  were  operating  as  described. 
The  system  development  and  maintenance  methodologies  estab- 
lished by  ASB  appear  logical  and  compare  to  industry  standards. 
ASB  management  has  a  process  in  place  to  periodically  update 
these  methodologies.   Users  of  the  systems  we  reviewed  at  ASB 
are  satisfied  with  the  development  and  maintenance  processes, 
and  believe  the  systems  are  beneficial. 


System  maintenance,  for  the  systems  we  reviewed,  appears  to  be 
completed  in  a  timely  manner.   ASB  staff  appear  to  have  back- 
grounds, experience,  and  skills  necessary  to  develop  and  main- 
tain automated  systems.   In  addition,  staff  are  kept  up-to-date 
on  appropriate  technologies. 

We  did  note  one  area  where  improvements  could  be  made.  The 
following  section  discusses  post-implementation  review  at  ASB, 
and  includes  our  recommendation  for  improvement. 


Post-Implementation 
Review 


ASB  provides  system  development  services  to  state  agencies. 
Once  a  system  is  implemented  and  accepted  by  an  agency,  ASB 
is  no  longer  responsible  for  the  system.   If  an  agency  does  not 
specifically  request  a  post-implementation  review  from  ASB,  an 
evaluation  of  the  development  process  is  not  completed. 
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The  Office  of  the  Legislative  Auditor  conducted  an  Electronic 
Data  Processing  Audit  of  the  Systems  Development  Bureau,  now 
the  Application  Services  Bureau  (ASB),  and  released  a  report  in 
1984.   A  recommendation  was  made  to  establish  a  post-imple- 
mentation review  process.   To  date,  ASB  management  has  not 
implemented  any  formal  post-implementation  review. 

Development  standards  suggest  a  post-implementation  review  as 
the  final  phase  of  a  system  development  life  cycle  methodology. 
A  post-implementation  review  can  be  used  to  evaluate  the 
effectiveness  of  the  development  process.   The  review  deter- 
mines whether  original  system  objectives  and  requirements  were 
met.   Post-implementation  reviews  also  provide  an  opportunity 
for  assessing  customer  satisfaction.   If  an  agency  is  dissatisfied 
with  ASB's  system  development  efforts,  a  post-implementation 
review  would  enable  the  agency  to  convey  its  concerns.   This 
would  help  ASB  management  identify  deficiencies  in  its 
methodology. 

The  other  two  agencies  included  in  our  review  have  established 
post-implementation  review  as  part  of  the  system  development 
process.   A  post-implementation  review  of  TEAMS  (The 
Economic  Assistance  Management  System)  conducted  by 
management  of  the  Department  of  Social  and  Rehabilitation 
Services  identified  several  problems  with  the  system.  These 
problems  were  brought  to  the  TEAMS  contractor  and  changes 
were  made  to  correct  problems.   As  a  result,  TEAMS  is  a  better 
system.   Without  this  review,  TEAMS  problems  may  not  have 
been  identified. 

ASB  is  not  getting  formal  feedback  on  its  system  development 
process,   if  problems  occur  as  a  result  of  deficiencies  in  ASB 
development  techniques,  future  development  projects  may  have 
similar  problems.   Finally,  if  agencies  are  not  satisfied  with  ASB 
services,  future  contracts  may  not  be  considered. 

To  increase  the  effectiveness  of  current  methodologies  and 
ensure  system  quality,  ASB  management  should  develop  and 
implement  policies  for  reviewing  systems  and  agency  satisfac- 
tion.  According  to  management,  staffing  priorities  and  customer 

Page  21 


Chapter  III 

Department  of  Administration 


budget  limitations  have  prevented  development  of  a  post-imple- 
mentation review  process.  This  type  of  review  would  be  part  of 
the  development  service  provided  by  ASB.  ASB  charges  fees  for 
services  so  agency  development  costs  would  increase.  A  brief 
review  could  limit  increased  costs.  Management  agrees  with  our 
findings  and  plans  to  develop  policy  for  providing  post- 
implementation  review  as  an  optional  service  to  state  agencies. 


Recommendation  #1 

We  recommend  the  Application  Services  Bureau  implement 
post-implementation  review  as  part  of  its  methodology  to 
ensure  system  requirements  are  met  and  to  evaluate 
customer  satisfaction. 
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Introduction 


This  chapter  provides  background  information  on  system  devel- 
opment and  maintenance  activities  within  the  Operations  and 
Technology  Division  (OTD)  at  the  Department  of  Social  and 
Rehabilitation  Services  (SRS).   It  also  discusses  specific  issues 
related  only  to  OTD  identified  during  our  performance  audit. 


Operations  and  Tech- 
nology Division  (OTD) 


SRS  recently  completed  a  reorganization.   During  our  audit,  the 
Office  of  Management  Analysis  and  Systems  was  responsible  for 
system  development  and  maintenance  for  the  department.  The 
recent  reorganization  created  a  new  division,  the  Operations  and 
Technology  Division,  with  responsibility  for  system  development 
and  maintenance. 


Two  bureaus  within  the  OTD  provide  system  development  and 
maintenance  services  for  the  department.  The  Internal  Main- 
frame Systems  Bureau  (IMSB)  is  responsible  for  mainframe 
development  and  maintenance.   The  Microcomputer  Applications 
Bureau  (MAB)  handles  microcomputer  based  development  and 
maintenance.  The  MAB  was  implemented  in  August  1991.   The 
following  figure  shows  OTD's  current  organization. 
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Figure  5 
Operations  and  Technology  Division  Organization 
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Program 
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Bureau 


Fiscal 
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Source:  Compiled  by  the  Office  of  the  Legislative  Auditor 
front  department  records. __ 


There  are  14  full-time  equivalent  (FTE)  positions  within  OTD 
responsible  for  system  development  and  maintenance.   This 
includes  two  bureau  chiefs,  two  special  project  directors,  two 
supervisors,  and  various  levels  of  Information  Systems  Specialist 
(ISS)  positions. 

Currently,  the  OTD  has  three  system  maintenance  projects  under 
contract  with  the  private  sector.  The  OTD  has  one  system  devel- 
opment project  under  contract  at  this  time.   The  IMSB  has  no 
development  projects  and  eighteen  support/maintenance  projects 
in  progress.  The  MAB  has  four  development  projects  in 
progress  and  seven  support/maintenance  projects.   Department 
personnel  indicate  approximately  90  to  95  percent  of  staff 
activity  involves  system  maintenance.  Recently,  all  large  devel- 
opment projects  have  been  contracted  out  to  the  private  sector. 
During  the  past  two  fiscal  years,  one  contracted  and  three 
internal  system  development  projects  were  completed. 

Table  4  details  expenses  for  OTD  for  the  past  three  fiscal  years. 
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Table   4 


Operations  and  Technology  Division 
System  Development/Maintenance  Expenses 


Mainframe 
Microcomputer 
Contracted* 
TOTALS 


FY   1990-91 

$       401,580 

0 

9,781,083 

$10,182,663 


FY   1991-92 

$     434,763 

130,293 

7,973,413 

$8,538,469 


FY   1992-93 
$     435,333 
207,742 

7,835,661 
$8,478,736 


•Includes  TEAMS,  SEARCHS,  and  MMIS 

Source:  Operations  and  Technology  Division  records. 


OTD  System  Development 


The  OTD  recently  formed  a  standing  committee  to  review 
system  development  and  maintenance  policies  and  standards. 
The  committee  proposed  a  new  system  development  methodology 
which  OTD  management  reviewed  and  approved.   OTD's  new 
system  development  life  cycle  consists  of  the  following  five  main 
phases:  1)  conceptual,  2)  definition,  3)  design,  4)  implemen- 
tation, and  5)  evaluation/acceptance.   The  development  process 
starts  with  a  request  from  a  department  employee  with  proper 
authority  for  requesting  development  and  maintenance  services. 


Conceptual  Phase 


The  conceptual  phase  includes  a  preliminary  evaluation  of  an 
idea  for  automation.   A  statement  of  need  or  a  problem  state- 
ment is  developed  during  this  phase.   Tasks  also  include  an 
initial  cost  benefit  analysis  and  an  impact  analysis. 


Definition  and  Design 


The  second  phase  requires  identification  of  system  functional 
requirements.   Tasks  include  identifying  current  system  func- 
tions, new  system  requirements,  and  resource  requirements.   A 
rough  project  management  plan  for  the  next  two  phases  is 
developed. 


In  the  next  phase  comprehensive  program  and  design  specifica- 
tions are  developed.  Tasks  include  designing  files  and  database 
structures,  documenting  various  procedures,  and  developing 
project  plans  for  testing,  training,  and  implementation.   The 
design  phase  also  includes  coding  and  testing  programs,  testing 
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procedures  and  the  entire  system  using  test  data,  and  initiating 
the  implementation/conversion  plan. 


Implementation  and  Eval- 
uation/Acceptance 


During  the  implementation  phase,  the  system  is  integrated  into 
day-to-day  operations.  Tasks  include  generating  user  docu- 
mentation, identifying  and  training  users,  and  creating  an  imple- 
mentation schedule.   The  final  phase  includes  an  evaluation  of 
the  benefits  of  the  system,  and  a  formal  acceptance  of  the  system 
from  management  and  user  staff. 


OTD  System  Maintenance 


The  system  maintenance  process  also  starts  with  a  request  from  a 
department  employee  with  proper  authority.   Maintenance 
includes  any  activities  required  to  keep  a  system  operational  and 
responsive.   Services  provided  include:  1)  production  recovery, 
2)  maintenance,  and  3)  enhancement.   Production  recovery  is 
correction  of  problems  to  ensure  proper  system  operation. 
Maintenance  and  enhancements  include  system  changes  to 
maintain  consistent  operation  or  strengthen  the  system  in  some 
fashion.   Staff  correct  problems  and  complete  system  changes  as 
necessary  or  requested  using  procedures  similar  to  those  used  for 
development. 


OTD  Contracted  Services 


SRS  contracts  for  development  and  maintenance  services  for 
larger,  more  complex  systems.   SRS  recently  contracted  for 
development  and  maintenance  of  TEAMS  (The  Economic  Assis- 
tance Management  System)  and  SEARCHS  (System  for  the 
Enforcement  And  Recovery  of  Child  Support).   The  department 
has  contracts  for  development  and  maintenance  services  for 
MMIS  (Montana  Medicaid  Information  System)  and  the  General 
Relief  Assistance  and  Project  Work  Program  System. 


We  reviewed  department  policy  for  contracting  system  develop- 
ment and  maintenance  services  to  determine  if  current  proce- 
dures assure  system  needs  are  met  in  the  most  efficient  and 
effective  manner.  The  following  sections  summarize  our 
observations. 
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Contracting  Procedures 


Management  considers  the  size  of  a  project  and  the  complexity 
of  a  system  to  determine  whether  or  not  to  contract  with  the 
private  sector  for  development  and  maintenance  services.   Mana- 
gement's philosophy  for  system  maintenance  is  to  contract  with 
the  entity  who  develops  a  system  for  a  minimum  of  three  years 
after  implementation.   Management  believes  it  is  critical  to  a 
successful  project  to  have  expertise  from  the  start,  and  to  keep 
that  expertise  until  the  system  is  operating  smoothly. 


First,  the  department  completes  an  internal  requirements  analysis 
to  determine  system  needs.   An  RFP  (request  for  proposal)  is 
then  developed  and  sent  out  to  potential  contractors.  The 
department  also  surveys  other  states  for  similar  systems  already 
in  operation.   At  the  same  time  the  department  obtains  input 
from  vendors.   In  addition,  federally  funded  projects  require 
development  of  an  Advanced  Planning  Document. 

Information  from  all  sources  is  compiled  and  an  evaluation  of 
RFP  responses  is  made.  The  department  develops  RFP  evalua- 
tion criteria  which  must  be  met  before  a  contractor  is  considered 
for  selection.  Each  RFP  is  scored  according  to  the  evaluation 
criteria.  The  department  selects  the  low-bid  from  qualified 
finalists,  then  negotiates  a  contract. 


TEAMS  Contract 


Our  audit  included  a  review  of  the  contract  for  TEAMS  and 
TEAMS  documentation.  Our  review  of  the  TEAMS  contract 
indicates  various  controls  over  system  development  and  mainte- 
nance. TEAMS  has  a  full-time  contract  manager  responsible  for 
day-to-day  operations.  The  development  contract  required 
reports  for  various  phases  of  the  project.   Department  manage- 
ment reviewed  and  approved  each  report.  All  reports  for 
TEAMS  development  were  completed  and  the  majority  met 
initial  time  schedules  set  by  the  contractor. 


There  are  performance  criteria  in  the  TEAMS  maintenance 
contract  which  must  be  met.  The  department  receives  monthly 
reports  from  the  contractor  and  holds  monthly  status  meetings 
regarding  TEAMS  performance.   A  committee  made  up  of 
department  management  and  contractor  personnel  reviews 
system  change  requests  from  TEAMS  users  and  establishes 
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priorities.   The  contractor  completes  and  implements  all  changes 
to  the  system. 

The  department  completed  a  survey  of  TEAMS  users.   As  part  of 
our  review,  we  also  contacted  several  TEAMS  users  and  obtained 
input  on  TEAMS  operations.   The  responses  we  received  were 
similar  to  those  compiled  by  the  department.   Overall,  users 
appear  to  like  TEAMS  and  believe  it  helps  them  complete  their 
jobs  more  efficiently. 

In  addition,  the  Office  of  the  Legislative  Auditor  recently 
completed  an  Electronic  Data  Processing  (EDP)  audit  of  TEAMS 
(93DP-31).   EDP  audit  staff  believe  TEAMS  meets  its 
established  objectives  and  is  operating  as  intended. 

TEAMS  Costs  According  to  OTD  management,  quality  system  development  and 

maintenance  is  expensive.   TEAMS  cost  approximately 
$10.4  million  to  develop.   The  state  funded  approximately 
$1.6  million  of  the  total  cost.   The  department's  contract  for 
facilities  management  (maintenance)  of  TEAMS  is  over 
$2.8  million  per  year.  This  does  not  include  costs  incurred  for 
operations  at  the  Information  Services  Division,  Department  of 
Administration. 

Facilities  management  includes,  among  other  items,  mainte- 
nance, support,  and  enhancements  required  to  keep  TEAMS 
current  with  state  program  requirements  and  technical  environ- 
ment.  For  example,  if  food  stamp  requirements  change,  the 
contractor  is  responsible  for  modifying  TEAMS  to  meet  new 
program  requirements.   Contract  staffing  levels  require  a  mini- 
mum of  14  professional  staff  and  6  qualified  computer  program- 
mers and  programmer  analysts  on  site.   In  summary,  it  appears 
the  contract  process  used  by  SRS  has  controls  built  in  to  ensure 
department  needs  are  met. 
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OTD  Related  Issues 


Introduction 


During  our  review  at  OTD,  we  identified  areas  where  system 
development  and  maintenance  activities  were  operating  as 
described.   System  development  and  maintenance  methodologies 
have  been  established  by  management  and  compare  to  industry 
standards.   Users  of  the  systems  we  reviewed  at  OTD  are  satis- 
fied with  the  development  and  maintenance  processes,  and 
believe  the  systems  are  beneficial. 


System  maintenance,  for  the  systems  we  reviewed,  appears  to  be 
completed  in  a  timely  manner.   Staff  appear  to  have  experience 
and  skills  necessary  to  develop  and  maintain  automated  systems. 

We  did  note  one  area  where  improvements  could  be  made.   The 
following  section  discusses  training  at  OTD  and  includes  our 
recommendation  for  improvement. 


Training 


As  part  of  our  audit,  we  reviewed  training  provided  to  OTD 
system  development  and  maintenance  personnel.  Overall,  we 
found  the  department  provides  only  minimal  training  for  these 
staff.  SRS  does  not  have  a  formal  training  program,  nor  does  it 
formally  track  employee  training.   Staff  receive  some  training 
which  appears  appropriate.   However,  the  amount  of  training 
received  does  not  appear  to  keep  staff  up-to-date  on  relevant 
data  processing  technology. 


We  developed  criteria  to  evaluate  the  types  and  amounts  of 
training  received  by  staff.   We  created  a  matrix  based  on  general 
job  responsibilities  for  personnel  in  ISS,  supervisory,  and  man- 
agement positions,  and  total  hours  of  training  received.   Type  of 
training  related  directly  to  job  responsibilities.   Amounts  of 
training  were  ranked  as  low  (less  than  10  hours  a  year),  medium, 
or  high  (more  than  40  hours  a  year).   Five  of  eight  personnel 
files  examined  had  no  documentation  indicating  an  adequate 
amount  of  training,  based  on  the  above  criteria.   For  example, 
one  ISS  position  file  had  no  documentation  of  development  or 
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maintenance  related  training  in  1992.  Four  supervisory/manage- 
ment position  files  had  no  documentation  of  management  related 
training  in  1992. 

One  of  the  key  components  in  the  management  of  personnel  is 
the  provision  of  training.   Training  is  provided  to  improve  or 
enhance  employees'  abilities  to  perform  job  duties  and  can 
improve  employee  morale. 

Without  a  formal  training  program,  data  processing  staff  have 
less  formal  access  to  changing  standards  and/or  up-to-date 
development  and  maintenance  technologies.   Lack  of  training 
may  cause  staff  to  use  ineffective  or  out-dated  procedures 
without  being  aware  of  the  problem.   As  a  result,  system  quality 
may  be  jeopardized  which  could  lead  to  problems  with  future 
system  operations. 

OTD  management  does  not  analyze  staff  needs  to  determine  if 
training  is  necessary.  Supervisors  we  talked  with  said  staff  could 
use  more  technical  training.   All  staff  we  talked  with  believe 
they  are  not  kept  up-to-date  on  current  technologies. 

According  to  management,  the  training  budget  does  not  have 
enough  funds  to  provide  ongoing  training.   However,  according 
to  Statewide  Budgeting  and  Accounting  System  (SBAS)  informa- 
tion, OTD  expended  approximately  $10,700  for  training,  includ- 
ing technical  publications,  in  fiscal  year  1991-92.   OTD 
expended  only  about  $3,100  for  training  in  fiscal  year  1992-93, 
of  which  approximately  $1,100  was  unrelated  to  system  develop- 
ment and  maintenance.   According  to  OTD  management,  the 
department  scaled  down  its  training  budget  in  fiscal  year  1992- 
93.   The  OTD  requested  funding  for  training  for  the  1995  bien- 
nium  based  on  expenditures  during  fiscal  year  1991-92. 

In  comparison,  the  Application  Services  Bureau  (ASB),  Depart- 
ment of  Administration,  expended  approximately  $14,800  for 
training  in  fiscal  year  1992-93.   ASB  has  established  an  ongoing 
training  program  for  staff  and  continues  to  analyze  employee 
training  needs. 
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We  believe  OTD  management  should  identify  training  needs  and 
implement  a  formal  employee  training  program  to  meet  these 
needs.  Formal  training  should  help  increase  employee  efficiency 
and  effectiveness.   As  a  result,  the  quality  of  system  develop- 
ment and  maintenance  should  increase.  This  will  provide 
customers  increased  assurance  of  quality  systems. 


OTD  management  indicates  training  is  contingent  on  funding. 
Current  funding  will  only  allow  implementation  of  a  limited 
training  plan.   Management  is  aware  of  deficiencies  and  is 
reviewing  the  training  program  at  the  Applications  Services 
Bureau,  Department  of  Administration,  to  help  identify  possible 
solutions.  Staff  will  attend  computer  training  this  fall,  and  plan 
to  train  other  department  staff.   In  addition,  OTD  staff  are 
developing  a  system  for  a  division  within  the  department  to 
track  staff  training.   OTD  plans  to  utilize  this  system  after 
implementation. 


Recommendation  #2 

We  recommend  the  Operations  and  Technology  Division: 

A.  Analyze  staff  training  needs. 

B.  Implement  a  training  program  to  meet  identified 
needs. 
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Introduction 


This  chapter  provides  background  information  on  system  devel- 
opment and  maintenance  activities  within  the  Department  of 
Labor  and  Industry  (DOLI).   It  also  discusses  a  specific  issue 
related  only  to  DOLI  identified  during  our  performance  audit. 


Department  of  Labor 
and  Industry  (DOLI) 
Reorganization 


The  Department  of  Labor  and  Industry  recently  completed  an 
extensive  reorganization  as  the  result  of  a  task  force  recommen- 
dation.  While  completing  our  performance  audit,  the  Office  of 
Information  Services  (OIS)  provided  system  development  and 
maintenance  services  for  the  department.   Our  work  concen- 
trated on  OIS  operations.   As  a  result  of  the  department's 
reorganization,  OIS  is  defunct.   OIS  system  development  and 
maintenance  staff  who  were  dedicated  to  specific  division  opera- 
tions were  decentralized  to  those  divisions.   Remaining  OIS  staff 
were  reorganized  under  the  new  Information  Services  Bureau, 
Centralized  Services  Division.  According  to  management,  staff 
within  the  Centralized  Services  Division  will  coordinate  system 
development  and  maintenance  activities  for  the  department. 


Currently,  three  divisions  within  DOLI  are  responsible  for 
system  development  and  maintenance:  1 )  Job  Service,  2) 
Unemployment  Insurance,  and  3)  Centralized  Services.   Staff 
from  various  sections  within  these  three  divisions  complete 
system  development  and  maintenance  activities.  The  following 
figure  shows  DOLI's  current  organization. 
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Figure  6 

Department  of  Labor  and  Industry  Organization 
(partial  -  as  of  9-93) 
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Source:  Carpi  led  by  the  Office  of  the  Legislative  Auditor 
from  department  records. 


There  are  19  full-time  equivalent  (FTE)  positions  involved  with 
system  development  and  maintenance  at  DOLL   Of  these,  6  FTE 
work  within  the  Job  Service  Division,  6  FTE  work  within  the 
Unemployment  Insurance  Division,  4  FTE  work  within  the 
Centralized  Services  Division,  and  the  remaining  3  FTE  work 
within  the  Employment  Relations  Division.   In  addition,  3  FTE 
within  Centralized  Services  are  involved  with  system  operations. 

At  the  time  of  our  audit,  OIS  had  five  development  projects  and 
six  support/maintenance  projects  in  progress.   In  addition,  OIS 
had  one  contract  with  the  private  sector  for  system  support. 
During  fiscal  years  1989-90  through  1991-92,  OIS  completed 
seven  development/maintenance  projects. 

Expenses  for  OIS  for  the  past  two  fiscal  years  are  as  follows. 
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Table  5 


Office  of  Information  Services 
System  Development/Maintenance  Expenses 


Personal  Services 
Operating  Expenses* 
Equipment 
TOTALS 


FY  1991-92 

$666,241 

207,689 

19,004 

$892,934 


FY  1992-93 

$693,086 

155,172 

19,978 

$868,236 


♦Includes  contracted  services  of  $37,532  and  $48,568,  respectively. 
Source:  Office  of  Information  Services  records. 


DOLI  System  Development 


OIS's  system  development  life  cycle  consisted  of  the  following 
six  phases:  1)  project  definition,  2)  analysis,  3)  design,  4)  coding, 
5)  testing,  and  6)  implementation.  This  methodology  was 
developed  over  the  past  three  years.  The  development  process 
starts  with  a  request  from  a  user  (a  department  employee  with 
proper  authority  for  requesting  development  and  maintenance 
services). 


Project  Definition 


In  the  first  phase,  project  definition,  a  decision  is  made  on  the 
feasibility  of  the  project.   Activities  include  defining  project 
goals  and  scope,  developing  measurable  objectives,  and  defining 
a  tentative  plan. 


Analysis  and  Design 


The  second  phase  involves  analysis  of  current  system  operations 
and  new  user  needs.   Activities  include  gaining  an  understanding 
of  what  the  user  wants  in  the  new  system,  defining  functional 
requirements,  completing  a  cost  benefit  analysis,  and  starting  a 
test  plan.   In  the  next  phase  high  level  and  low  level  designs  are 
developed.   High  level  design  activities  include  developing 
diagrams  and  specifications,  determining  general  file  structures 
and  input/output  requirements,  and  continuing  to  develop  a  test 
plan  and  project  work  plan.   Low  level  design  activities  include 
developing  more  detailed  diagrams,  defining  data  requirements, 
designing  screens  and  reports  according  to  the  high  level  design, 
and  determining  whether  the  design  meets  goals,  scope,  objec- 
tives, and  requirements. 
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Coding,  Testing,  and 
Implementation 


The  next  two  phases,  coding  and  testing,  provide  a  working 
computer  system  based  on  requirements  from  previous  phases. 
Activities  for  these  phases  include  coding  programs,  testing 
programs,  procedures  and  the  entire  system  using  test  data, 
deriving  expected  test  results,  and  determining  if  project 
objectives  and  requirements  have  been  met.   The  final  phase, 
implementation,  includes  developing  an  implementation  schedule 
and  training  schedule,  completing  a  follow-up  survey  with  the 
user,  and  obtaining  formal  acceptance  and  project  sign-off. 


DOLI  System  Maintenance 


The  system  maintenance  process  also  starts  with  a  request  from  a 
user.   Maintenance  includes  any  activities  required  to  keep  a 
system  operational  and  responsive.   Services  provided  include:  1) 
production  recovery,  2)  maintenance,  and  3)  enhancement. 
Production  recovery  is  correction  of  problems  to  ensure  proper 
system  operation.   Maintenance  and  enhancements  include 
system  changes  to  maintain  consistent  operation  or  strengthen 
the  system  in  some  fashion.   Staff  correct  problems  and  complete 
system  changes  as  necessary  or  requested  using  procedures 
similar  to  those  used  for  development. 


DOLI  Contracted  Services 


We  did  not  review  procedures  for  contracted  services  at  OIS. 
Contractors  have  been  used  in  the  past,  but  were  not  hired 
recently  to  provide  system  development  or  maintenance  services. 
There  is  one  exception;  OIS  contracted  with  the  private  sector 
for  support  of  a  system  which  the  same  company  developed 
several  years  ago.   DOLI  does  not  utilize  development  or  mainte- 
nance services  provided  by  the  Information  Services  Division, 
Department  of  Administration. 
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DOLI  Related  Issues 


Introduction 


During  our  review  at  DOLI,  we  noted  areas  where  system 
development  and  maintenance  activities  were  operating  as 
described.   DOLI's  system  development  and  maintenance 
methodologies  appear  logical  and  compare  to  industry  standards. 
Users  of  the  systems  we  reviewed  are  satisfied  with  the 
development  and  maintenance  processes,  and  believe  the  systems 
are  beneficial. 


System  maintenance,  for  the  systems  we  reviewed,  appears  to  be 
completed  in  a  timely  manner.  Staff  appear  to  have  necessary 
skills  and  experience  to  develop  and  maintain  automated  systems. 
Staff  are  also  kept  up-to-date  on  appropriate  technologies. 

We  did  note  one  area  where  improvement  is  needed.   No  one 
entity  is  responsible  for  data  processing  activity  within  the 
department.  Staff  within  several  divisions  of  the  Department  of 
Labor  and  Industry  are  responsible  for  system  development  and 
maintenance.  The  following  section  discusses  this  issue  and 
includes  our  recommendation  for  improvement. 


Separation  of  Develop- 
ment/Maintenance Staff 


As  mentioned  previously,  staff  within  three  divisions  of  DOLI 
conduct  system  development  and  maintenance.   During  our 
review,  we  contacted  personnel  from  these  divisions  and 
obtained  input  on  development  and  maintenance  responsibilities. 
While  our  work  in  these  divisions  was  limited,  we  did  note  no 
single  entity  has  input/control  over  development  and  mainte- 
nance work  conducted  by  personnel  within  DOLI. 


The  Administrative  Support  Bureau  within  the  Job  Service 
Division  is  responsible  for  both  mainframe  and  microcomputer 
system  development  and  maintenance.  Staff  complete  work  as 
needed  and  requested  by  Job  Service  personnel.   In  addition, 
staff  within  the  Apprenticeship  and  Training  Bureau  are  respon- 
sible for  maintaining  systems  controlled  by  the  federal  govern- 
ment. According  to  bureau  management,  staff  only  complete 
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system  development  and  maintenance  activities  as  needed  to 
provide  data  services. 

Two  sections  within  the  Unemployment  Insurance  (UI)  Division 
are  also  responsible  for  system  development  and  maintenance:    1) 
Administrative  Support,  and  2)  Research  and  Analysis.  Staff 
within  these  two  sections  complete  work  as  needed  and  requested 
by  UI  system  users.   Separation  of  responsibilities  in  this  division 
is  similar  to  the  Job  Service  Division.   One  section  completes 
department  system  development  and  the  other  section  is  respons- 
ible for  maintenance  of  federal  government  systems. 

In  addition,  staff  within  the  Information  Services  Bureau, 
Centralized  Services  Division,  provide  system  development  and 
maintenance  services  for  divisions  without  data  processing 
support.  This  bureau  is  comprised  of  staff  who  were  not 
decentralized  from  OIS. 

There  are  differences  in  the  way  system  development  and  main- 
tenance is  conducted  within  different  divisions  of  the  depart- 
ment.  Each  division  follows  its  own  system  development  and 
maintenance  methodologies.   Department  management  has 
established  general  guidelines  for  department  personnel,  but 
there  is  no  formal  process  to  ensure  all  personnel  follow  similar 
standards. 

Development  of  standards  and  coordination  of  system  develop- 
ment and  maintenance  activities  should  help  reduce  possible 
duplication  of  effort.   In  addition,  department  standards  and 
coordination  should  help  maximize  consistency  and  system 
compatibility,  which  should  help  increase  information  sharing 
capabilities. 

The  department's  current  funding  structure  contributes  to  the 
current  situation.   Each  division  receives  funding  directly. 
Division  management  allocates  funding  to  its  programs.  The 
department's  centralized  services  function  receives  funding  from 
the  divisions  it  serves.  Each  division,  with  the  exception  of  the 
Centralized  Services  Division,  operates  individually.   As  a  result, 
no  single  entity  controls  or  oversees  department  data  processing 
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activities  to  ensure  system  development  and  maintenance  is 
consistent  and  does  not  duplicate  other  department  activity. 

We  believe  the  department  should  define  roles  and  responsi- 
bilities for  all  DOLI  data  processing  personnel.   One  entity 
should  be  given  responsibility  for  direction  and  control  of  data 
processing  in  the  department,  including  development  of  depart- 
ment-wide standards.   This  should  help  increase  interdepart- 
mental communication  and  coordination. 

According  to  management,  a  goal  of  DOLI  is  to  be  more  focused 
department-wide.   The  department  plans  to  form  a  core  group 
made  up  of  information  technology  managers  from  several 
divisions  within  DOLI.  This  core  group  will  be  responsible  for 
coordinating  data  processing  activity  within  the  department. 


Recommendation  #3 

We  recommend  the  Department  of  Labor  and  Industry 
assign  direction  and  control  of  data  processing  activity  to 
one  entity  within  the  department. 
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Introduction 


This  chapter  discusses  system  development  and  maintenance 
concerns  common  to  all  three  agencies  included  in  our  audit:  1) 
the  Application  Services  Bureau  (ASB);  2)  the  Operations  and 
Technology  Division  (OTD);  and  3)  the  Office  of  Information 
Services  (OIS).   Although  OIS  is  now  defunct,  staff  and  responsi- 
bilities still  remain  within  the  Department  of  Labor  and  Industry 
(DOLI).   Some  staff  from  OIS  were  decentralized  and  the 
remaining  staff  were  reorganized  under  the  new  Information 
Services  Bureau.   Three  divisions  in  the  department  are  now 
responsible  for  system  development  and  maintenance.   As  a 
result,  our  findings  and  recommendations  are  still  applicable  to 
DOLI. 


We  noted  areas  where  system  development  and  maintenance 
activities  were  operating  as  documented  and  described  to  us. 
The  most  important  of  these  relates  to  system  development  and 
maintenance  methodologies.   All  three  agencies  included  in  our 
review  have  established  standards  for  developing  and  maintain- 
ing automated  systems.   We  also  reviewed  qualifications  of  a 
sample  of  staff  within  the  three  agencies  in  our  review.   We 
found  personnel  have  necessary  skills  to  develop  and  maintain 
automated  systems.   The  following  sections  summarize  our 
findings,  including  recommendations  for  improvement.   Our 
recommendations  relate  to  increasing  the  effectiveness  of  current 
operations. 


Coordination  in  System 
Development 


As  part  of  our  audit,  we  asked  data  processing  personnel  in  the 
agencies  we  reviewed  what  communication  and  coordination 
they  have  with  other  state  agencies.   We  found  very  little  com- 
munication and  coordination  between  state  agencies  regarding 
system  development. 


Section  1-0232,  Montana  Operations  Manual  (MOM),  states 
interdepartmental  sharing  should  be  considered  when  developing 
systems.  This  section  of  the  MOM  further  states,  "All  agency 
development  procedures  should  incorporate  a  review  of  existing 
software  to  determine  if  an  alternative  to  custom  programming 
would  be  cost  beneficial." 
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Systems  developed  by  agencies  to  increase  efficiency  and  effec- 
tiveness of  operations,  no  matter  how  large  or  small,  may  be 
beneficial  to  other  state  agencies.   However,  potential  benefits 
may  never  be  realized  without  some  form  of  communication  or 
coordination.   As  a  result,  potential  cost  savings,  increases  in 
efficiency,  and  other  benefits  to  agencies  and  the  state  may  be 
lost.   In  addition,  there  may  be  some  duplication  of  effort  by 
agencies  developing  similar  systems. 

For  example,  during  House  Joint  Resolution  (HJR)  48  subcom- 
mittee meetings,  officials  from  the  judiciary  branch  said  if 
system  planning  and  development  were  coordinated,  SRS  could 
have  incorporated  child  support  activities  of  the  courts  into  its 
child  support  enforcement  system  (SEARCHS).   Instead,  these 
two  agencies  have  two  different  systems  monitoring  similar 
activities.   At  some  point  these  two  systems  need  to  be  integrated 
to  increase  child  support  enforcement  coordination. 

Section  1-0232,  MOM,  states  cooperation  with  and  consideration 
of  systems  in  other  state  agencies  can  reduce  the  cost  of 
information  needed  by  multiple  agencies.  The  HJR  48 
subcommittee  recommended  all  state  agencies  take  into  account 
interdepartmental  coordination  in  system  planning  and  feasibility 
studies. 

The  Department  of  Administration  has  established  MOM  policies 
for  state  agencies  to  follow  for  planning  and  managing  system 
development  projects.   It  is  the  responsibility  of  each  state 
agency  to  develop  and  implement  system  development  methodol- 
ogies, which  incorporate  MOM  policies.   This  should  help 
increase  possibilities  for  realizing  benefits  to  agencies  and  the 
state. 
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Recommendation  #4 

We  recommend  the  Application  Services  Bureau,  the  Oper- 
ations and  Technology  Division,  and  the  Department  of 
Labor  and  Industry  incorporate  Montana  Operations 
Manual  policy  regarding  coordination  and  information 
sharing  into  current  system  development  methodologies. 


System  Testing  and  Test  Testing  is  a  requirement  in  all  industry  standards  regarding 

Documentation  system  development  and  maintenance,  including  standards 

developed  by  the  three  agencies  in  our  review.   Generally,  there 
are  four  types  of  testing:  1)  unit  or  program,  2)  integration  or 
module,  3)  system,  and  4)  acceptance.   Unit  testing  refers  to 
testing  of  individual  programs.   Integration  testing  involves 
testing  of  programs  as  a  group  or  subsystem  within  a  system. 
Prior  to  acceptance  testing,  all  programs  and  subsystems  are 
assembled  and  tested  as  a  working  system.  This  is  called  the 
system  test.  The  final  level  of  testing  is  the  acceptance  test. 
Acceptance  testing  is  usually  completed  by  the  customer.  The 
acceptance  test  ensures  the  system  is  ready  to  be  put  into 
production.   Any  of  the  four  types  of  testing  can  be  used  during 
system  development  and/or  maintenance. 

Standards  also  exist  regarding  test  documentation.   According  to 
A  Standard  for  Auditing  Computer  Applications  (William  E. 
Perry),  if  testing  is  properly  performed,  a  test  plan,  test  results, 
and  a  test  report  should  be  available.  In  addition,  The  Complete 
Guide  to  Software  Testing  (William  Hetzel)  defines  critical 
control  elements  to  be  a  test  plan,  a  record  of  testing,  and  a 
record  of  test  results.  A  test  plan  provides  an  opportunity  for 
independent  review  and  approval  of  proposed  testing.  Docu- 
mentation of  test  activities  enables  management  to  monitor 
project  progress.   It  also  provides  supervisors  a  means  for 
reviewing  and  approving  staff  work.   Management  can  use  test 
results  to  analyze  system  capabilities  and  deficiencies,  and 
determine  if  a  system  or  system  change  is  ready  for 
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implementation.  Test  results  also  provide  user  management  with 
information  needed  to  determine  whether  to  accept,  modify,  or 
reject  a  system  or  system  change. 


ASB  Testing  Policy 


ASB's  current  policy  is  general  regarding  testing.   However, 
policy  does  require  a  test  plan,  a  log  of  testing  activity,  and  a 
record  of  test  results.   ASB  has  developed  draft  policy  on  testing 
which  provides  more  detail  on  required  documentation. 


OTD  Testing  Policy 


OTD's  current  policy  states  ".  .  .in  order  to  ensure  the  reliability 
of  all  production  systems,  test  plans  must  be  formulated  for 
newly  developed  systems  and  for  all  modifications  to  existing 
systems."  Department  policy  requires  development  of  a  general 
test  plan  followed  by  development  of  a  detailed  test  plan  when 
developing  a  new  system.   Policy  also  requires  development  of  a 
maintenance  test  plan  for  all  system  maintenance  and 
enhancements.   According  to  policy,  test  plans  must  be 
comprehensive  enough  to  ensure  a  system  is  working  properly. 
The  development  test  plan  should  include  unit  testing,  system 
testing,  and  acceptance  testing.   The  maintenance  test  plan 
should  include  a  list  of  programs  and/or  procedures  to  be  tested, 
anticipated  results,  and  a  date  indicating  when  anticipated  results 
were  achieved.   Policy  requires  completed  test  plans  to  be 
reviewed  by  the  bureau  chief  to  ensure  all  steps  were  followed 
and  all  anticipated  results  were  achieved. 


DOLI  Testing  Policy 


DOLI's  current  policy  requires  a  request  for  programming  (RFP) 
file  folder  for  each  request  received.   All  documentation  for 
each  request  is  maintained  in  the  RFP  file.  The  RFP  file  must 
contain,  among  other  items,  a  test  plan,  test  data,  and  test  results. 
Policy  requires  ongoing  development  of  a  test  plan,  careful 
selection  of  test  data,  and  documentation  of  test  results.   DOLI 
policy  requires  similar  documentation  during  system 
maintenance. 
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Testing  Policies  Not  Our  review  indicates  noncompliance  with  current  testing  policies 

Followed  in  all  three  agencies.   At  ASB,  we  found  the  Regents'  Employee 

Reporting  System  (RERS)  has  no  documentation  related  to  test- 
ing, no  test  plan,  test  scripts,  or  test  results.   Test  scripts  are 
being  maintained  for  the  Public  Employees'  Retirement  Division 
New  Active  System,  but  a  test  plan  had  not  been  developed  as  of 
our  review. 


For  OTD,  the  Internal  Mainframe  Systems  Bureau  just 
completed  a  rewrite  of  the  Low  Income  Energy  Assistance 
Program  (LIEAP)  System.   A  major  rewrite  of  the  Medicaid  Paid 
Claims  System  was  also  completed  in  April  1991.   The  Micro- 
computer Applications  Bureau  is  currently  rewriting  the  Client 
Database  and  down-sizing  the  system  from  the  mainframe  to  the 
microcomputer  environment.   No  documentation  related  to 
testing  was  observed  for  these  systems;  no  test  plans  or  test 
results.   Limited  documentation  of  user  testing  was  observed  for 
these  systems.   Of  52  total  maintenance  requests  reviewed,  only  1 
request  had  documentation  related  to  testing.   The  Client  Data- 
base rewrite  project  is  still  in  the  early  stages  of  development; 
however,  at  the  time  of  our  audit,  project  schedules  did  not 
require  test  results.  OTD's  new  system  development  life  cycle 
now  requires  test  plans  and  test  results. 

Within  DOLI,  staff  completed  a  project  replacing  local  job 
service  office  IBM  8100  equipment  and  systems  and  consolidat- 
ing all  offices  onto  the  state's  mainframe  computer.  Staff  also 
implemented  a  Safety  Licensing  System  for  the  Research,  Safety, 
and  Training  Division.   No  documentation  related  to  testing  was 
observed  for  either  of  these  systems;  no  test  plans,  test  data,  or 
test  results.   We  did  observe  documentation  related  to  acceptance 
testing  for  these  systems.   In  addition,  we  reviewed  maintenance 
documentation  for  the  Eagle  System  and  the  Benefits  and  Tax 
systems.   Of  the  30  total  requests  reviewed,  12  did  not  have  a 
test  plan.   Test  data  and  test  results  were  observed  for  all  but  1 
of  the  30  requests. 

Although  it  appears  staff  within  these  three  agencies  conduct 
some  testing  during  system  development  and  maintenance, 
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appropriate  documentation  of  testing  is  not  formally  maintained. 
Staff  keep  some  personal  files  of  test  documentation  like  screen 
printouts,  source  code  listings,  and  test  data  during  unit  testing 
for  some  systems,  but  formal  documentation  is  not  always  kept 
as  part  of  system  documentation.   Due  to  the  lack  of  appropriate 
test  documentation,  there  is  no  assurance  required  testing  was 
completed. 


Testing  in  Production 
More  Costly 


All  areas  critical  to  a  system  may  not  be  tested  if  test  plans  are 
not  developed,  reviewed,  and  approved.  The  system  design  may 
not  be  followed  and  system  specifications  may  not  be  met. 
Insufficient  testing  decreases  possibilities  for  identifying  system 
errors  and  deficiencies.  Testing  will  never  achieve  100  percent 
confidence,  but  failure  to  adequately  test  significant  system 
components  could  create  problems  during  future  system  opera- 
tions.  According  to  industry  guidelines,  correcting  system  errors 
and/or  deficiencies  after  a  system  is  in  production  is  much  more 
time  consuming  and  costly  than  if  done  during  initial 
development. 


If  test  documentation  is  not  maintained  and  reviewed,  manage- 
ment cannot  properly  evaluate  system  and  staff  performance. 
Without  documentation  of  test  results,  management  also  cannot 
adequately  determine  whether  a  system  or  system  change  is 
ready  for  implementation. 

Several  factors  contribute  to  the  lack  of  test  documentation. 
Guidelines  on  test  documentation  content  and  retention,  and 
testing  procedures  are  not  specific.   Management  has  no  formal 
process  for  reviewing  test  documentation  to  determine  compli- 
ance with  standards.   Finally,  although  staff  and  management 
appear  to  recognize  the  importance  of  testing,  the  importance  of 
developing  test  plans  and  need  for  documentation  of  testing  is 
not  emphasized  by  management. 

Management  should  enforce  its  own  testing  requirements  to 
ensure  new  systems  and  revisions  function  properly.   Manage- 
ment should  also  enforce  its  own  documentation  requirements 
related  to  the  planned  approach,  and  the  extent  and  results  of 
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testing.   Documentation  should  include  details  of  planned  testing 
as  well  as  a  summary  of  test  results. 

ASB  plans  to  review  its  policies  and  procedures  and  more  clearly 
define  requirements.   OTD  management  stated  they  do  not 
formally  approve  test  documentation  due  to  limited  personnel 
resources.   OTD  plans  to  implement  a  change  control  document 
which  will  be  used  by  staff  to  document  testing  performed. 
DOLI  sees  room  for  improvement  in  documenting  system  testing 
and  believes  it  can  do  a  better  job  of  enforcing  policies. 


Recommendation  #5 

We  recommend  the  Application  Services  Bureau,  the  Oper- 
ations and  Technology  Division,  and  the  Department  of 
Labor  and  Industry: 

A.  Develop  more  specific  guidelines  on  testing  procedures 
and  test  documentation  content  and  retention. 

B.  Enforce  system  testing  and  test  documentation 
requirements. 


System  Documentation  System  documentation  is  critical  for  development  and  mainte- 

nance of  automated  systems.  Management,  programmers,  and 
customers  need  documentation  to  aid  communications  and  refer 
to  during  the  life  of  a  system.  Documentation  helps  reveal 
system  weaknesses  and  in  doing  so  improves  effectiveness. 
Accuracy  and  efficiency  of  measuring  project  progress  and 
success  will  improve  with  quality  documentation.   Without 
documentation,  anyone  involved  with  a  system  must  rely  on 
memory.  System  documentation  can  also  be  used  to  help  train 
new  staff  members,  both  programmer/analysts  and  system  users. 

As  part  of  our  audit,  we  compared  standards  of  the  three  depart- 
ments to  industry  standards  to  determine  the  appropriateness  of 
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current  system  documentation  standards.   We  also  reviewed 
documentation  for  several  systems  developed  and/or  maintained 
by  each  of  the  agencies  to  determine  compliance  with  depart- 
ment standards.  Current  documentation  standards  of  each 
department  are  comparable  to  industry  standards.   However,  we 
found  established  policies  and  procedures  are  not  followed. 


System  Documentation 
Required  for  ASB 


According  to  ASB's  technical  guide,  documentation  of  prelimi- 
nary analysis  will  include  project  scope  and  objectives,  business 
needs,  system  requirements,  cost/benefit,  a  recommendation  on 
how  to  proceed,  and  preliminary  plans  for  the  remainder  of  the 
project.  The  guide  refers  to  documentation  for  the  second  phase 
of  system  development  including  technical  documentation, 
system  software,  a  test  plan,  and  a  plan  for  training.   ASB 
standards  require  formal  concurrence  from  the  customer  that  the 
project  meets  the  scope  and  objectives.   Finally,  ASB's  guide 
requires  periodic  informal  internal  audits. 

ASB  standards  also  provide  general  guidelines  for  staff  to  follow 
when  conducting  system  maintenance.  Tasks  which  may  be 
performed  parallel  system  development  procedures.   One  of  the 
intended  objectives  is  to  assist  in  maintaining  the  quality  of  an 
existing  system  by  applying  appropriate  design  methods  to  each 
task  and  by  maintaining  documentation. 

ASB  guidelines  indicate  a  customer's  signature  should  always  be 
obtained  on  the  form  used  to  request  support  services.  Guide- 
lines detail  tasks  to  perform  when  providing  system  support. 
These  include  preparing  and  updating  documentation.  Guide- 
lines require  all  production  software  to  have  a  program  change 
log  within  the  source  code  identifying  the  service  request  form 
number,  date,  and  person  completing  the  change(s).  In  addition, 
documentation  requirements  include  a  listing  from  COMPAREX 
software  comparing  the  new  version  of  a  program  to  the  old 
version. 
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System  Documentation 
Required  for  OTD 


OTD  policy  states  procedures  and  standards  are  to  be  strictly 
followed  by  all  department  personnel.   The  standards  manual 
describes  the  purposes,  tasks,  staffing  needs,  and  products 
(deliverables)  for  each  phase  of  the  development  methodology. 
Policy  also  requires  a  post-implementation  review  after  a  system 
has  been  in  production  for  at  least  six  months.   According  to 
standards,  each  system  must  be  sufficiently  documented  to  allow 
maintenance  or  modification  without  the  assistance  of 
individuals  involved  in  the  original  development. 


OTD's  current  development  documentation  standards  require 
various  types  of  system  and  technical  documentation.   New  draft 
policy  also  mandates  additional  types  of  documentation. 
Required  documentation  includes  system  narratives,  flowcharts, 
program  definitions,  a  statement  of  need  or  initial  problem 
statement,  a  functional  requirements  document,  a  finalized  cost 
benefit  statement,  program  documentation,  and  an  implementa- 
tion plan  and  schedule. 

OTD's  standards  manual  provides  a  general  guideline  for  staff  to 
follow  when  conducting  system  support.   Any  modifications  to 
systems  must  adhere  to  the  same  standards  and  procedures  used 
for  system  development.  Procedures  describe  the  process  staff 
should  follow  when  completing  system  modifications  including 
required  documentation.   For  each  program  modification, 
standards  require  programs  to  contain  the  programmers  name,  a 
date,  the  change  request  number,  and  a  description  of  the  change 
made. 


System  Documentation 
Required  for  DOLI 


Current  DOLI  standards  require  documentation  of  automated 
information  systems  so  the  system  description  remains  current 
and  accurate.   Department  manuals  describe  the  mission, 
standards,  and  documentation  required  for  system  development 
and  maintenance. 


DOLI's  current  documentation  standards  require  source  code 
documentation,  a  documentation  checklist,  a  general  system 
overview,  technical  documentation,  and  minimum  RFP  file 
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folder  requirements.   RFP  file  requirements  include  the  RFP 
form,  project  scope  and  objectives,  time  and  cost  estimations, 
input  and  output  views,  test  documentation,  and  other  pertinent 
documents. 


Systems  Reviewed 


We  reviewed  development  documentation  for  the  following 
systems  at  ASB:  RERS  (Regent's  Employee  Reporting  System), 
and  the  Public  Employees'  Retirement  Division  (PERD)  New 
Active  System.   We  also  reviewed  maintenance  documentation 
for  RERS  and  the  State  Trust  Land  Management  System. 


We  reviewed  system  development  documentation  for  the  follow- 
ing three  internally  developed  and  maintained  systems  at  OTD: 
the  Low  Income  Energy  Assistance  Program  (LIEAP)  System, 
the  Medicaid  Paid  Claims  System,  and  the  Client  Database.   We 
also  reviewed  maintenance  documentation  for  the  LIEAP  and 
Medicaid  Paid  Claims  systems. 

Finally,  we  reviewed  system  development  documentation  for  the 
following  two  systems  at  DOLI:  the  8100  Project,  and  the  Safety 
Licensing  System.   We  also  reviewed  maintenance  documentation 
for  the  EAGLE  System  and  the  Benefits  and  Tax  systems. 


Not  All  Required  Docu- 
mentation Maintained 


It  does  not  appear  all  required  system  documentation  is  main- 
tained for  system  development  and  maintenance  projects.  In 
addition,  documentation  is  not  consistent  between  projects.   The 
general  system  development  life  cycle  is  followed,  but  specific 
requirements  followed  depend  on  the  people  involved  with  the 
project. 

Our  review  of  systems  shows  that  technical  documentation 
appears  to  be  kept  during  system  development.   However,  system 
documentation  is  not  consistent.   Aside  from  technical  aspects, 
documentation  varies  from  project  to  project.  There  are  differ- 
ences even  in  how  system  documentation  is  organized.   Notes, 
letters,  and  memos  are  inconsistent  in  format,  content,  and 
retention.  System  documentation  is  also  not  formally  reviewed 
for  compliance  with  department  standards. 
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There  was  no  documentation  of  formal  acceptance  of  RERS. 
For  RERS  and  the  PERD  New  Active  System,  only  one  memo 
noted  management  approval  of  development.   For  RERS,  4  of  22 
service  request  forms  lacked  formal  approval.   Several  changes 
were  not  indicated  in  RERS  source  code  documentation. 
Finally,  12  of  41  forms  did  not  have  documented  management 
approval.   Forms  also  have  no  indication  of  management  review. 

Management  approval  was  obtained  for  the  first  phase  of  devel- 
opment of  the  Client  Database  project.   In  addition,  all  service 
request  forms  were  signed  by  both  data  processing  and  user 
management.   However,  there  were  no  cost/time  estimations  or 
target  dates  for  8  of  40  Medicaid  Paid  Claims  system  changes.   In 
addition,  7  of  40  changes  were  not  indicated  in  Medicaid  Paid 
Claims  source  code  documentation. 

Formal  management  approval  was  not  observed  for  either  the 
8100  Project  or  the  Safety  Licensing  System.   Only  1  of  the  30 
maintenance  requests  reviewed  had  formal  management 
approval.   There  were  no  project  scope  and  objectives  for  5  of 
the  30  total  RFP  files  reviewed.   We  also  reviewed  the  program 
change  log  for  the  EAGLE  System  and  found  5  of  20  changes 
were  not  indicated  in  source  code  documentation. 

Lack  of  Documentation  a  Lack  of  documentation  is  a  common  problem  associated  with 

Common  Problem  development  and  maintenance  of  automated  systems.   According 

to  industry  experience  and  agency  staff,  system  documentation  is 
typically  completed  after  the  system  and/or  change  is  opera- 
tional.  Users  drive  the  process  and  are  more  interested  in  an 
operational  system  than  adequate  documentation.   As  a  result, 
system  documentation  is  given  a  low  priority.   According  to 
management,  lack  of  resources  and  time  causes  system 
documentation  to  suffer. 

Inconsistency  and/or  lack  of  documentation  can  create  problems 
and  inefficiencies  in  operations.  Inconsistent  documentation  can 
cause  time  delays  when  staff  switch  from  one  project  to  another. 
Lack  of  documentation  can  increase  problems  and  costs 
associated  with  system  operation.  Without  complete  documenta- 
tion, system  changes  became  more  difficult.  Staff  have  limited 
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knowledge  of  system  design  and  operation.   This  can  create 
additional  problems  when  system  changes  are  completed  which 
unknowingly  affect  other  areas  of  the  system. 

Although  current  documentation  standards  at  the  three  agencies 
appear  to  provide  a  basis  for  developing  and  maintaining  quality 
systems,  current  system  documentation  standards  are  not 
followed  and  documentation  is  not  completed  consistently 
between  projects.   Management  should  emphasize  the  importance 
of  system  documentation  to  the  overall  project.   In  addition, 
management  should  ensure  complete  documentation  exists,  as 
appropriate,  for  all  system  development  and  maintenance 
projects. 

Management  at  ASB  will  review  policies  with  the  intent  of  more 
consistently  maintaining  documentation  from  project  to  project. 
OTD  policies  and  procedures  are  being  reviewed  and  modified  to 
reflect  current  operations.   In  addition,  OTD  established  a 
documentation  system  in  1992  to  provide  more  uniform  docu- 
mentation.  DOLI  management  encourages  staff  to  tailor  proce- 
dures depending  on  project  size,  scope,  and  need.   DOLI  man- 
agement will  clarify  this  authority  with  written  guidelines. 


Recommendation  #6 

We  recommend  the  Application  Services  Bureau,  Opera- 
tions and  Technology  Division,  and  the  Department  of 
Labor  and  Industry: 

A.  Emphasize  the  importance  of  system  documentation. 

B.  Ensure  complete  documentation  exists  for  ail  system 
development  and  maintenance  projects. 
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Management 
Information 


ASB  management  uses  the  project  control  software  package 
PC/70  to  monitor  system  development  and  maintenance  projects. 
A  LOTUS  spreadsheet  is  used  to  estimate  project  costs. 
Personnel  at  ASB  recently  conducted  a  review  of  project  man- 
agement software  and  purchased  a  package  to  aid  in  project 
management.   Based  on  our  review,  it  appears  this  new  software 
package  could  help  strengthen  management  information  at  ASB. 


Lack  of  Management 
Information  Systems  - 
OTD  &  DOLI 


Neither  OTD  nor  DOLI  have  management  information  systems 
to  monitor  system  development  and  maintenance.   Several 
project  management  tools  are  available  to  staff  at  OTD  and 
DOLI,  including  Microsoft  Project,  but  are  not  used  consistently 
for  project  monitoring. 


Various  reports  tracking  work  requests,  staff  time,  estimated 
costs,  and  project  status  are  compiled  by  each  department.   Some 
quarterly  reports  are  also  generated.   Other  management  infor- 
mation is  kept  informally  or  generated  as  needed.  Management 
at  OTD  and  DOLI  do  not  formally  track  actual  costs  or  time 
frames  and  compare  them  with  project  estimations. 

Accurate,  timely,  and  appropriate  management  information  is 
essential  for  providing  effective  control.   A  successful  project 
relies  on  sound  management  of  staff  and  project  activities. 
Controls,  including  management  information,  must  be  in  place  to 
enable  management  to  evaluate  progress  and  make  necessary 
corrections.   Industry  standards  suggest  collection  and  analysis  of 
numerous  types  of  management  information  as  part  of  project 
control.   System  development  and  maintenance,  operations,  and 
personnel  are  main  areas  of  concentration. 

If  management  information  is  inaccurate  or  appropriate  infor- 
mation is  not  collected,  program  operations  could  suffer.   Lack 
of  management  information  could  cause  management  to  make 
uninformed  decisions  which  negatively  affect  staff  and/or  the 
quality  of  systems. 
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No  Formal  Measurement 
of  Performance 


ASB  management  is  aware  of  insufficiencies  with  project  cost 
estimation  and  planning,  and  recently  purchased  a  new  project 
management  software  system.   However,  ASB  management  does 
not  measure  bureau  performance  or  compare  bureau  results  to 
established  goals  and  objectives.   It  appears  OTD  and  DOLI 
management  compile  some  project  statistics,  but  it  does  not 
appear  management  evaluates  project  activities  and  costs  to 
identify  concerns. 

There  is  no  formal  process  in  place  for  evaluating  performance 
in  completing  work  requests  on  time.   Management  information 
relates  specifically  to  number  of  work  requests  and  project 
status.   Management  does  not  analyze  time  or  costs  associated 
with  system  development  and  maintenance. 

Management  at  these  agencies  should  gather  information  for  use 
in  measuring  system  and  staff  productivity.   Project  estimations 
such  as  costs  and  time-lines  should  be  compared  to  actual  results 
to  determine  accuracy.  Other  types  of  information  should  also 
be  collected  and  analyzed  to  help  determine  both  system  and 
staff  productivity. 

Management  should  review  the  usefulness  of  current  information 
available  to  it.  Current  project  management  software  should  be 
reviewed  for  adequacy  of  monitoring  projects  and  generating 
management  information.   Once  reviewed,  necessary  action 
should  be  taken  to  ensure  adequate  information  is  available  to 
make  informed  decisions.   These  actions  will  help  improve 
project  and  staff  management. 

Management  at  ASB  agrees  with  our  recommendations  and  plans 
to  purchase  two  additional  "modules"  for  its  new  project  man- 
agement software  system.  OTD  management  recently  developed 
a  project  time  reporting  system  to  help  gather  information.  In 
addition,  OTD  management  has  started  to  analyze  requirements 
for  management  information  and  plans  to  further  develop  its 
management  information  system  as  department  priorities  allow. 
DOLI  management  also  agrees  with  our  recommendations  and 
plans  to  continue  to  develop  and  implement  new  ideas  for  its 
management  information  system. 
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Recommendation  #7 

We  recommend  the  Application  Services  Bureau,  the  Oper- 
ations and  Technology  Division,  and  the  Department  of 
Labor  and  Industry: 

A.  Analyze  the  usefulness  of  current  management  infor- 
mation for  project  and  staff  management. 

B.  Develop  a  project  management  system  which  collects 
and  maintains  appropriate  management  information 
needed  for  use  in  directing  and  evaluating  perfor- 
mance. 


Performance  Appraisals  During  our  audit  we  examined  employee  personnel  files  to 

determine  whether  performance  appraisals  are  completed  in 
accordance  with  the  Administrative  Rules  of  Montana  (ARM). 
Performance  appraisals  are  completed  at  ASB.   We  found  only 
one  of  eight  OTD  personnel  files  reviewed  had  a  current  perfor- 
mance appraisal.  Three  OTD  files  reviewed  had  no  performance 
appraisals,  although  two  of  the  three  files  had  completed  pre- 
appraisals.   We  found  four  of  eight  Department  of  Labor  and 
Industry  (DOLI)  personnel  files  reviewed  did  not  have  a  current 
performance  appraisal.   All  four  DOLI  personnel  files  lacking 
performance  appraisals  were  management  positions. 

Section  2.21.6411,  ARM,  states  performance  appraisals  are  to  be 
completed  for  each  full-time  and  part-time  position  during 
established  appraisal  periods  of  not  more  than  one  year's  dura- 
tion.  Appraisals  provide  direction  and  motivation  for  employees. 
Areas  requiring  corrective  action  or  additional  training  can  be 
identified  during  the  appraisal  process. 

Failure  to  complete  periodic  performance  appraisals  makes  it 
difficult  for  management  to  monitor  and  document  employee 
productivity  and  program  effectiveness.   Lack  of  performance 
appraisals  also  limits  department  management's  ability  to  ensure 

Page  55 


Chapter  VI 
Common  Concerns 


program  intent  is  met.   Without  performance  appraisals,  staff 
may  not  receive  information  on  productivity  which  could 
negatively  affect  morale.   A  performance  appraisal  system  could 
help  management  identify  and  correct  potential  inadequacies. 

It  appears  department  administration  placed  a  low  priority  on 
completion  of  performance  appraisals.   As  a  result,  department 
management  has  not  placed  emphasis  on  completion  of  annual 
evaluations. 

Department  management  should  complete  timely  appraisals  of  all 
staff.   Periodic  appraisals  will  increase  department  management's 
ability  to  evaluate  employee  performance.   An  appraisal  system 
will  provide  employees  with  formal  recognition  for  productive 
activity  and  help  identify  areas  for  improvement. 

Performance  appraisals  are  not  a  priority  at  OTD.  OTD  manage- 
ment believes  time  spent  on  appraisals  cannot  be  recovered 
through  increased  employee  efficiency.  The  Commissioner  of 
the  Department  of  Labor  and  Industry  believes  in  the  perfor- 
mance appraisal  process  and  has  taken  action  to  update  and 
standardize  a  performance  appraisal  system. 


Recommendation  #8 

We  recommend  the  Operations  and  Technology  Division, 
and  the  Department  of  Labor  and  Industry  implement 
formal  procedures  for  completing  timely  performance 
appraisals  of  all  staff. 


System  Development 
and  Maintenance 
Standards 
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Management  activities  involve  planning,  organizing,  directing, 
and  controlling.   Established  standards  (policies  and  procedures) 
provide  general  controls  over  system  development  and  mainte- 
nance activity.  Controls  direct  data  processing  staff  on  proce- 
dures which  must  be  followed  when  developing  new  systems 
and/or  changing  existing  systems.  Controls  also  strengthen 
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management's  control  over  program  operations  and  help  assure 
continuity  of  services  as  staffing  changes  occur. 


No  Formal  Process  -  OTD 


Standards  must  be  carefully  planned  and  continually  monitored 
to  ensure  effective  implementation.   One  responsibility  of 
management  is  to  monitor  the  performance  of  program  activities. 
Management  must  determine  whether  controls  function  properly. 
Monitoring  standards  will  enable  management  to  detect  and 
correct  deficiencies  in  order  to  improve  program  operations, 
including  system  development  and  maintenance  procedures. 


OTD  management  has  no  formal  process  for  reviewing  and 
updating  standards.   According  to  management,  standards  are 
only  updated  every  so  often,  maybe  every  five  years.  Staff  are 
given  opportunity  to  make  suggestions  and  recommend  changes, 
but  there  is  no  formal  process. 

Although  OTD's  standards  manual  was  updated  in  June  1991, 
certain  sections  are  still  outdated.  For  example,  system  develop- 
ment procedures  refer  to  the  SRS  Policy  Committee  which  is 
obsolete.  Project  control  procedures  are  referred  to  in  system 
support  procedures  but  have  not  been  established.  In  addition, 
draft  policy  was  recently  developed  and  has  some  variances  from 
current  procedures  relating  to  the  development  methodology  and 
required  documentation.  We  believe  new  draft  policy  is  inade- 
quate because  it  requires  less  deliverables  than  industry  standards 
and  current  OTD  policy. 


No  Formal  Process  -  DOLI 


DOLI  also  has  no  formal  process  for  reviewing  and  updating 
system  development  and  maintenance  standards.   The  Job 
Service  Division  has  not  formally  established  microcomputer 
standards,  but  staff  indicate  industry  standards  will  be  used  as  a 
base.  The  Unemployment  Insurance  Division  also  has  not 
established  formal  guidelines  due  to  the  diversity  of  its  projects. 
Centralized  Services  Division  staff  will  follow  standards 
developed  by  the  now  defunct  Office  of  Information  Services 
(OIS).  In  addition,  staff  decentralized  from  OIS  to  the  Job 
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Service  and  Unemployment  Insurance  divisions  will  continue  to 
follow  OIS  standards. 

Standards  can  become  outdated  quickly  in  the  data  processing 
field.   Lack  of  formal  and  up-to-date  standards  can  cause  incon- 
sistencies in  operations  and  impede  staff  performance.   Without 
a  formal  standards  monitoring  process,  the  status  of  system 
development  and  maintenance  functions  may  not  be  known. 
Staff  also  may  not  be  developing  and  maintaining  systems  in  the 
most  efficient  and  effective  manner.   If  potential  problems  are 
not  identified,  staff  will  continue  to  follow  improper  procedures. 
The  longer  problems  exist,  the  more  severe  the  results. 

Historically,  agencies  only  establish  department-wide  guidelines 
for  issues  which  are  common  to  all  department  personnel. 
General  data  processing  functions  are  usually  covered  under 
department  policy.  Specific  standards  are  usually  left  up  to  each 
individual  program.   Management  believes  operations  are  func- 
tioning properly  and  current  standards  are  sufficient  to  meet 
needs.   However,  our  review  indicates  need  for  more  specific 
direction  from  department  management  regarding  system  devel- 
opment and  maintenance  standards. 

Due  to  short  time  lapses  between  data  processing  industry 
changes,  department  management  needs  to  formalize  a  process  to 
review  and  update  standards  on  a  regular  basis.   Management 
must  implement  a  process  to  periodically  review  standards.  This 
will  help  ensure  system  development  and  maintenance  standards 
are  kept  up-to-date.   It  should  also  increase  consistency  between 
divisions  and  increase  the  possibility  of  data  processing  staff 
participating  in  other  division  projects. 

OTD  management  is  currently  developing  a  new  system 
development  life  cycle  and  plans  to  revise  standards  after  the 
new  methodology  has  been  finalized.   In  addition,  OTD  manage- 
ment plans  to  establish  a  process  for  updating  standards.   The 
Department  of  Labor  and  Industry  also  agrees  with  our  recom- 
mendations.  As  mentioned  in  Chapter  V,  DOLI  plans  to  create  a 
core  group  consisting  of  information  technology  managers  from 
divisions  responsible  for  system  development  and  maintenance. 
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This  core  group  will  be  responsible  for  ensuring  communication 
between  divisions  remains  open  and  system  development  and 
maintenance  standards  are  consistent  department-wide. 


Recommendation  #9 

We  recommend  the  Operations  and  Technology  Division, 
and  the  Department  of  Labor  and  Industry  establish  a 
formal  process  for  monitoring  and  periodically  updating 
system  development  and  maintenance  standards. 
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Methodologies  are  in 
Place 


This  audit  focused  on  system  development  and  maintenance 
activities  within  three  agencies.   Overall,  we  found  each  of  the 
operations  reviewed  have  both  system  development  and  system 
maintenance  methodologies  in  place.   This  provides  a  base  for 
development  and  maintenance  of  automated  systems.  The 
methodologies  established  within  each  of  the  departments  are 
generally  comparable  to  industry  standards.   Users  of  the  systems 
we  reviewed  indicated  overall  satisfaction  with  processes  used  by 
the  three  agencies. 


Methodologies  are  not 
Always  Followed 


While  we  found  areas  where  system  development  and  mainte- 
nance activities  were  operating  as  documented  and  described  to 
us,  we  also  found  areas  where  the  methodologies  are  not 
followed.   We  noted  some  common  concerns  with  various  aspects 
of  system  development  and  maintenance  activities.  These 
include: 

—     system  testing  and  test  documentation. 

--     system  documentation. 

--     system  development  and  maintenance  standards. 

The  potential  effects  resulting  from  these  concerns  are  twofold. 
First,  systems  may  not  be  as  cost  effective  as  possible.  Secondly, 
system  deficiencies  may  not  be  identified. 


Improved  Management 
Controls  Needed 


During  our  review,  we  also  identified  the  following  management 
controls  which  could  be  improved: 

—  management  information. 

—  performance  appraisals. 

Improvements  in  management  controls  will  provide  increased 
assurance  to  management  and  the  state  that  automated  systems 
are  developed  and  maintained  effectively. 


Formal  system  development  and  maintenance  methodologies 
provide  check-points  for  management  and  staff  to  measure 
progress  and  success.   We  believe  implementation  of  our 
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recommendations  will  increase  the  effectiveness  of  departmental 
operations.   In  addition,  implementation  will  provide  department 
management  and  the  state  with  increased  assurance  that  auto- 
mated systems  are  developed  and  maintained  in  the  most  effi- 
cient and  economic  manner  possible. 
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DEPARTMENT  OF  ADMINISTRATION 

DIRECTOR'S  OFFICE 


MARC  RACICOT,  GOVERNOR 


MITCHELL  BUILDING 


STATE  OF  TvONTANA 


(406) 444  2032 
FAX:  444-2812 


November    19,     19  93 


PO  BOX  200101 
HELENA.  MONTANA  S9620-0101 


!'    |?-;  \z 


Jim  Pellegrini  ,  r--:.'vp»" 

Deputy  Legislative  Auditor 
Office  of  Legislative  Auditor 
Room  135,  State  Capitol 
Helena,  MT   59620 

Dear  Jim: 

The  department's  response  to  the  October,  1993  Performance  Audit 
Report  of  Automated  System  Development  and  Maintenance  is  as 
follows : 

RECOMMENDATION  #1 

WE  RECOMMEND  THE  APPLICATION  SERVICES  BUREAU  IMPLEMENT  P0ST- 
IMPLEMENTATION  REVIEW  AS  PART  OF  ITS  METHODOLOGY  TO  ENSURE  SYSTEM 
REQUIREMENTS  ARE  MET  AND  TO  EVALUATE  CUSTOMER  SATISFACTION. 

Response 

The  department  concurs  with  this  recommendation.  ASB  will  include 
a  post -implementation  review  task  as  part  of  its  methodology. 

Scheduled  Implementation  Date:   03/01/94 

RECOMMENDATION  #2 

N/A 

RECOMMENDATION  #3 

N/A 

RECOMMENDATION  #4 

WE  RECOMMEND  THE  APPLICATION  SERVICES  BUREAU,  THE  OPERATIONS  AND 
TECHNOLOGY  DIVISION,  AND  THE  DEPARTMENT  OF  LABOR  AND  INDUSTRY 
INCORPORATE  MONTANA  OPERATIONS  MANUAL  POLICY  REGARDING  COORDINATION 
AND  INFORMATION  SHARING  INTO  CURRENT  SYSTEM  DEVELOPMENT 
METHODOLOGIES. 
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Response 

The  department  concurs  with  this  recommendation.  ASB  will  include 
in  its  design  methodology  a  task  to  insure  interdepartmental 
sharing  has  been  considered  for  each  project  and  the  customer 
agency  has  been  encouraged  to  communicate  its  plans  with  other 
state  agencies  through  ITMG  and  ITAC. 

Scheduled  Implementation  Date:   03/01/94 

RECOMMENDATION  #5 

WE  RECOMMEND  THE  APPLICATION  SERVICES  BUREAU,  THE  OPERATIONS  AND 
TECHNOLOGY  DIVISION,  AND  THE  DEPARTMENT  OF  LABOR  AND  INDUSTRY: 

A.  DEVELOP  MORE  SPECIFIC  GUIDELINES  ON  TESTING  PROCEDURES 
AND  TEST  DOCUMENTATION  CONTENT  AND  RETENTION. 

B.  ENFORCE    SYSTEM    TESTING    AND    TEST    DOCUMENTATION 
REQUIREMENTS . 

Response 

The  department  concurs  with  this  recommendation.  ASB  will  review 
all  of  its  policies  and  practices  on  testing  and  test  documentation 
with  the  intent  of  more  clearly  defining  testing  requirements.  The 
objective  of  this  review  will  be  to  define  a  model  set  of  testing 
requirements  that  will  be  tailored  to  each  project.  To  the  extent 
required  by  the  results  of  this  review,  ASB  will  enforce  testing 
and  test  documentation  requirements. 

Scheduled  Implementation  Date:   09/01/94 

RECOMMENDATION  #6 

WE  RECOMMEND  THE  APPLICATION  SERVICES  BUREAU,  OPERATIONS  AND 
TECHNOLOGY  DIVISION,  AND  THE  DEPARTMENT  OF  LABOR  AND  INDUSTRY: 

A.  EMPHASIZE  THE  IMPORTANCE  OF  SYSTEM  DOCUMENTATION. 

B.  ENSURE  COMPLETE  DOCUMENTATION  EXISTS  FOR  ALL  SYSTEM 
DEVELOPMENT  AND  MAINTENANCE  PROJECTS. 

Response 

The  department  concurs  with  this  recommendation.  ASB  will  review 
all  of  its  policies  and  practices  on  technical  documentation  with 
the  intent  of  clarifying  the  importance  of  system  documentation  in 
ASB's  work  methods,  training,  and  written  procedures  that  identify 
documentation  requirements. 

ASB  will  also  establish  procedures  that  ensure  system  documentation 
requirements  are  being  met  by  the  bureau. 
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Scheduled  Implementation  Date:   05/01/94 

RECOMMENDATION  #7 

WE  RECOMMEND  THE  APPLICATION  SERVICES  BUREAU,  THE  OPERATIONS  AND 
TECHNOLOGY  DIVISION,  AND  THE  DEPARTMENT  OF  LABOR  AND  INDUSTRY: 

A.  ANALYZE  THE  USEFULNESS  OF  CURRENT  MANAGEMENT  INFORMATION 
FOR  PROJECT  AND  STAFF  MANAGEMENT. 

B.  DEVELOP  A  PROJECT  MANAGEMENT  SYSTEM  WHICH  COLLECTS  AND 
MAINTAINS  APPROPRIATE  MANAGEMENT  INFORMATION  NEEDED  FOR 
USE  IN  DIRECTING  AND  EVALUATING  PERFORMANCE. 

Response 

The  department  concurs  with  this  recommendation.   Specifically: 

1)  Two  additional  "modules"  of  the  Project  Workbench  software 
have  been  acquired  and  are  being  incorporated  into  ASB's  work 
methods  as  part  of  the  ongoing  effort  to  improve  the  process 
of  collecting  and  evaluating  project  management  information. 

2)  The  current  practice  of  using  PC/70,  Project  Workbench, 
and  other  management  tools  will  be  reviewed  to  determine  their 
usefulness.  The  review  will  also  determine  whether  or  not 
current  practices  should  be  augmented  with  a  process  for 
identifying  summary  data  that  will  become  a  regular  part  of 
management  information. 

Scheduled  Implementation  Date:   07/01/94 

RECOMMENDATION  #8 

N/A 

RECOMMENDATION  #9 

N/A 

Sincerely, 

Lois  Menzies 
Director 

cc :   Mike  Trevor,  Administrator,  Information  Services  Division 
Jeff  Brandt,  Chief,   Application  Services  Bureau 
Sharon  Gorie,  Supervisor,  Computing  Policy  and  Development 
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MARC  RACICOT 
GOVERNOR 


DEPARTMENT  OF  ;rV 

SOCIAL  AND  REHABILITATION  SERVICES  ;L 


PETER  S    BLOUKE,  PhD 
DIRECTOR 


STATE  OF  MONTANA 


P.O.  BOX  4210 
HELENA.  MONTANA  S9604-4210 


November  18,  1993 


Jim  Pellegrini 

Deputy  Legislative  Auditor 

Performance  Audit 

Office  of  Legislative  Auditor 

State  Capitol 

Helena,  MT  59620 

Dear  Mr.  Pellegrini: 

Enclosed  is  the  Department  response  to  the  Performance  Audit 
Report  of  October,  1993  on  Automated  System  Development  and 
Maintenance.  Most  of  our  concerns  and  corrective  actions  were 
included  in  the  final  report.  The  enclosure  completes  our 
response  and  needs  to  be  added  to  the  report. 

Please  contact  me  if  you  have  any  questions. 

Sincerely, 


nuoL«w\ 


Michael/G.    Billirjlgs 

Administrator 

Operations  and  Technology  Division 


enc , 
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DEPARTMENT  OF  SOCIAL  AND  REHABILITATION  SERVICES 

RESPONSE  TO 

OFFICE  OF  LEGISLATIVE  AUDITOR 

PERFORMANCE  AUDIT  REPORT 

AUTOMATED  SYSTEM  DEVELOPMENT  AND  MAINTENANCE 

October  1993 

The  following  is  the  Department  of  Social  and  Rehabilitation 
Services  (SRS)  response  to  the  Legislative  Auditors  Performance 
Audit  Report  on  Automated  System  Development  and  Maintenance  for 
the  Department  of  Administration,  Department  of  SRS  and  Department 
of  Labor  and  Industry. 

Chapter  IV  -  Department  of  SRS 

Legislative  Auditors  Recommendation  #2 

We  recommend  the  Operations  and  Technology  Division: 

A.  Analyze  staff  training  needs. 

B.  Implement  a  training  program  to  meet  identified  needs. 
SRS  Response 

The  Department  concurs  with  the  recommendation  and  is  imple- 
menting a  training  program  as  indicated  in  the  Performance 
Audit  Report. 

Chapter  VI  -  Common  Concerns 

Legislative  Auditors  Recommendation  #4 

We  recommend  the  Applications  Services  Bureau,  the  Operations 
and  Technology  Division  and  the  Department  of  Labor  and 
Industry  incorporate  Montana  Operations  Manual  policy  regard- 
ing coordination  and  information  sharing  into  current  system 
development  methodologies. 

SRS  Response 

The  Department  concurs  with  the  recommendation.  SRS  will 
continue  to  share  and  coordinate  information  system  technology 
with  other  Departments.  SRS  is  assisting  the  Department  of 
Family  Services  in  development  of  new  information  systems 
while  supporting  current  shared  information  systems.  The 
development  of  TEAMS  and  SEARCHS  was  coordinated  with  the 
Department  of  Administration.  The  Department  has  developed 
and  shared  with  other  departments  criteria  for  access  to 
automated  databases. 
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Legislative  Auditors  Recommendation  #5 

We  recommend  the  Application  Services  Bureau,  the  Operations 
and  Technology  Division  and  the  Department  of  Labor  and 
Industry: 

A.  Develop  more  specific  guidelines  on  testing  procedures 
and  test  documentation  content  and  retention. 

B.  Enforce  system  testing  and  test  documentation  reguire- 
ments. 

SRS  Response 

The  Department  concurs  with  the  recommendation.  The  SRS 
Information  Systems  Development  Methodology,  adopted  September 
1,  1993,  reguires  a  test  plan  and  retention  of  test  results 
for  systems  development  and  maintenance.  The  Department  will 
enforce  this  methodology  including  the  testing  provisions. 


Legislative  Auditors  Recommendation  #6 

We  recommend  the  Application  Services  Bureau,  Operations  and 
Technology  Division  and  the  Department  of  Labor  and  Industry: 

A.  Emphasize  the  importance  of  system  documentation. 

B.  Ensure  complete  documentation  exists  for  all  system 
development  and  maintenance  projects. 

SRS  Response 

The  Department  concurs  with  the  recommendation.  The  SRS 
Information  Systems  Development  Methodology  identifies 
documentation  reguirements  for  system  development  and  mainte- 
nance. 


Legislative  Auditors  Recommendation  #7 

We  recommend  the  Application  Services  Bureau,  the  Operations 
and  Technology  Division  and  the  Department  of  Labor  and 
Industry: 

A.  Analyze  the  usefulness  of  current  management  information 
for  project  and  staff  management. 

B.  Develop  a  project  management  system  which  collects  and 
maintains  appropriate  management  information  needed  for 
use  in  directing  and  evaluating  performance. 

SRS  Response 

The  Department  concurs  with  the  recommendation  and  is  develop- 
ing a  project  management  information  system  as  indicated  in 
the  Performance  Audit  Report. 

70 


Legislative  Auditors  Recommendation  #8 

We  recommend  the  Operations  and  Technology  Division  and  the 
Department  of  Labor  and  Industry  implement  formal  procedures 
for  completing  timely  performance  appraisals  of  all  staff. 

SRS  Response 

The  Department  concurs  with  the  recommendation.  Operations 
and  Technology  Division  management  will  emphasize  implementa- 
tion of  existing  SRS  policies  on  formal  performance  appraisals 
in  accordance  with  the  Administrative  Rules  of  Montana. 


Legislative  Auditors  Recommendation  #9 

We  recommend  the  Operations  and  Technology  Division  and  the 
Department  of  Labor  and  Industry  establish  a  formal  process 
for  monitoring  and  periodically  updating  system  development 
and  maintenance  standards. 

SRS  Response 

The  Department  concurs  with  the  recommendation.  An  Operations 
and  Technology  Division  committee  has  been  appointed  and  is 
meeting  to  review  information  systems  standards. 
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DEPARTMENT  OF  LABOR  AND  INDUSTRY 
COMMISSIONER'S  OFFICE 


MARC  RACICOT.  GOVERNOR 


P.030X  1728 


STATE  OF  MONTANA 


(406)444-3555 
FAX  (406)444-1394 


HELENA.  MONTANA   59624 


November  19,  1993 


Scott  A.  Seacat 

Legislative  Auditor 

Office  of  the  Legislative  Auditor 

State  Capitol 

Helena,  MT  59620 

Dear  Scott: 

Attached  is  our  response  to  your  final  report  on  the  audit  of  the  state  automated 
system  development  and  maintenance.  Copies  of  the  audit  report  you  provided  to  us 
are  also  enclosed. 

We  appreciate  the  cooperation  of  your  staff  in  taking  the  time  to  review  the  audit 
findings  with  us  in  detail.  The  report  is  very  helpful. 

The  department  administrators  and  I  will  be  happy  to  continue  working  with  you  and 
the  Legislative  Audit  Committee  to  answer  questions  about  the  audit  and  our 
response. 

Thanks  again. 


Sincerely, 


LAURIE  EKANGER 
Commissioner 


y^y 


LE:ka 
enclosures 


72 


Quality  Works 


Report  to  the  Legislative  Auditor 

on 
Automated  System  Development  and  Maintenance 

Prelude;  The  Department  of  Labor  and  Industry  (DLI)  recognized 
there  were  improvements  to  be  made  in  the  Department's  Information 
Services  and  in  computer  programs  systems  development.  A  task 
group  was  formed  to  survey  and  analyze  the  situation.  The 
comprehensive  Report  to  the  Commissioner  Concerning  Information 
Services,  dated  June  7,  1993  described  the  situation  and  outlined 
a  solution  strategy  of  decentralizing  certain  information  services 
and  retaining  other  information  services  in  a  centralized  arena 
(the  Information  Services  Bureau  of  the  Centralized  Services 
Division)  .  A  second  part  of  the  solution  was  to  form  a  DLI 
Information  Technology  Managers  Group  (composed  of  technical 
managers  representing  each  division)  which  advises  the 
Administrators  and  Commissioner  on  IT  issues. 

Recommendation  #3 

— We  recommend  that  DLI  assign  direction  and  control  of  data 

processing  activity  to  one  entity  within  the  Department. 

DLI  has  designated  ISB,  using  the  input  and  expertise  of  the 
Department  IT  manager's  group,  to  perform  the  function  of 
coordinating  data  processing  activity  in  DLI.  The  ISB  Bureau  Chief 
is  the  facilitator  of  this  group  and  coordinates  the 
recommendations  to  assure  direction  established  by  the  Commissioner 
and  Administrators  is  followed. 

Recommendation  #4 

— We  recommend  the  Application  Services  Bureau,  the  Operations  and 
Technology  Division,  and  the  Department  of  Labor  and  Industry 
incorporate  Montana  Operations  Manual  policy  regarding  coordination 
and  information  sharing  into  current  system  development 
methodologies . 

DLI  agrees.  The  Montana  State  Information  Technology 
Management  Group  (ITMG)  has,  in  the  July  14,  1993  charter,  this 
item  as  one  of  the  main  purposes  for  the  group.  The  Auditor's 
concern  is  that  the  function  is  not  being  carried  out;  DLI  will 
reguest  a  discussion  of  this  item  during  the  December  ITMG  meeting. 

R^oommendatign  #5 

— We  recommend  the  Application  Services  Bureau,  the  Operations  and 

Technology  Division,  and  the  Department  of  Labor  and  Industry: 

A.  Develop  more  specific  guidelines  on  testing  procedures  and 
test  documentation  content  and  retention. 

B.  Enforce  system  testing  and  test  documentation  requirements. 

DLI  agrees.  DLI  has  guidelines  on  testing  and  on  test 
documentation,  but  those  guidelines  are  generic,  mainframe- 
oriented,  and  somewhat  old.  Also,  compliance  with  those  guidelines 
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is  inconsistent.  The  DLI  IT  Managers  Group  will  rewrite  those 
guidelines  and  forward  them  to  the  Auditor's  office.  The  DLI  IT 
Managers  Group  will  be  responsible  for  the  implementation  of  the 
revised  policy  in  their  respective  areas.  The  DLI  Administrators 
will  enforce  this  policy  in  their  respective  areas.  Followup  and 
monitoring  will  be  done  by  ISB  Chief. 

Recommendation  #6 

— We  recommend  the  Application  Services  Bureau,  the  Operations  and 

Technology  Division,  and  the  Department  of  Labor  and  Industry: 

A.  Emphasize  the  importance  of  system  documentation. 

B.  Ensure  complete  documentation  exists  for  all  system 
development  and  maintenance  projects. 

DLI  agrees.  DLI  recognizes  the  importance  of  system 
documentation  for  all  projects.  The  DLI  IT  managers  will  be 
responsible  for  ensuring  that  all  systems  have  complete 
documentation.  The  DLI  Administrators  will  enforce  this  policy  in 
their  respective  areas.  Followup  and  monitoring  will  be  done  by 
ISB  Chief. 

Recommendation  #7 

— We  recommend  the  Application  Services  Bureau,  the  Operations  and 

Technology  Division,  and  the  Department  of  Labor  and  Industry: 

A.  Analyze  the  usefulness  of  current  management  information  for 
project  and  staff  management. 

B.  Develop  a  project  management  system  which  collects  and 
maintains  appropriate  management  information  needed  for  use  in 
directing  and  evaluating  performance. 

DLI  agrees.  First,  DLI  will  have  a  task  force  study  the 
usefulness  of  current  project  management  information.  The  results 
of  that  study  will  be  given  to  ISB  and  the  IT  managers,  who  will  be 
tasked  with  developing  an  appropriate  project  management  system. 
We  further  request  the  Auditor's  assistance  in  identifying  an 
existing  system  which  will  automatically  collect  this  data  for  all 
platforms,  rather  than  developing  home  made  systems  for  this 
function. 

Recommendation  #8 

— We  recommend  the  Operations  and  Technology  Division  and  the 
Department  of  Labor  and  Industry  implement  formal  procedures  for 
completing  timely  performance  appraisals  of  all  staff. 

DLI  agrees.  The  Department  has  a  policy  in  place  regarding 
this.  Furthermore,  all  administrators  now  have  performance 
objectives  in  place  which  require  that  position  descriptions  and 
performance  appraisals  be  in  place  and  updated  annually  for  all 
staff. 
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Recommendation  #9 

— We  recommend  the  Operations  and  Technology  Division  and  the 
Department  of  Labor  and  Industry  establish  a  formal  process  for 
monitoring  and  periodically  updating  system  development  and 
maintenance  standards. 

DLI  agrees.  The  DLI  IT  Managers  Group  will  establish  a  formal 
process  for  monitoring  and  updating  systems  standards.  The  DLI 
Administrators  will  oversee  this  process.  The  ISB  Chief  will 
coordinate  and  ensure  follow  up. 
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